Enhancing Feature Diversity via Spectral Regularization During Pretraining of Vision Transformers on Small Datasets

In computer vision, the pretraining–finetuning paradigm serves as a fundamental framework for training Vision Transformers (ViTs). However, training ViTs from scratch on small datasets without large-scale pretraining remains challenging. In this paper, we investigate the evolution of the Empirical Spectral Density (ESD) of ViTs during training and propose a regularization method based on the singular value spectrum. Specifically, we analyze how the implicit self-regularization of ESD of weights affects the feature space mapped by the weights, particularly the phenomenon of limited feature diversity under a high Signal-to-Noise Ratio (SNR). We then introduce an explicit regularization approach that jointly optimizes the Singular Value Entropy (SVE) and the spectral norm of the weight matrices, aiming to enhance both the feature SNR and diversity during pretraining. Experiments on multiple small datasets demonstrate that the proposed method improves the performance of ViTs and their variants in training from scratch scenarios.