Relating Model Performance to Embedding Distributions in Molecular Machine Learning

Choosing effective molecular representations remains a central challenge in molecular machine learning and is often addressed through costly trial-and-error. While model selection is typically guided solely by predictive performance, analyzing relationships between trained models can reveal additional structure missed by performance metrics. In terms of model similarity metrics, representational alignment techniques, such as centered kernel alignment (CKA), provide a principled framework for comparing models beyond their predictive performance. In this work, we show that representational alignment is fundamentally linked to performance differences between models. For linear regression, we theoretically show that alignment upper-bounds the achievable performance gaps. This result predicts an exclusion zone in which highly aligned models do not exhibit large performance differences, a phenomenon we empirically validate across 661 classification data sets. To make these insights actionable, we introduce the mean minimum class distance (MMCD), a straightforward data set-level statistic that predicts a data set's position in the alignment-performance difference space. Across 23 molecular representations and ten representative data sets, we find that data sets that produce highly aligned models tend to exhibit low MMCD, suggesting that alignment is strongly shaped by data set-specific structure. Overall, our results indicate that when alignment is low, exploring alternative representations is more likely to improve performance. In contrast, when alignment is high, gains are more effectively achieved by increasing the size of the training data.