Unifying Variables in Neural Scaling for General Audio Representations via Embedding Effective Rank

In neural scaling, scaling laws have profoundly shaped our understanding of model performance across easily scalable and expressible parameters. However, when scaling involves multiple parameters that present complex expressions or are difficult to express, it becomes challenging to formulate scaling functions that encompass such parameters. In this work, We found that in the audio domain, theembedding effective rank(RankMe) can serve as a unifying metric that enables label-free, information-theoretic quantification of audio embeddings, encapsulates the impact of diverse variables including masking rate, model size, data volume, computational budget, and architectural configurations on representation quality and demonstrates more significant effectiveness compared to traditional scaling laws. Our empirical findings reveal a consistent power-law relationship between RankMe and representation quality, suggesting that embedding effective rank serves as a reliable proxy for assessing model performance in audio representation learning. Based on RankMe as a label-free proxy metric, our work demonstrates its role as an early-stage monitoring tool. Furthermore, we provide an empirically validated framework for early screening of optimal models from multiple complex hyperparameter settings during pre-training, which saves computational resources.