Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings
Abstract
Log-likelihood vectors define a common space for comparing language models as probability distributions, enabling unified comparisons across heterogeneous settings.We extend this framework to training checkpoints and intermediate layers, and establish a consistent scale for KL divergence across pretraining, model size, random seeds, quantization, fine-tuning, and layers.Analysis of Pythia pretraining trajectories further shows that changes in loglikelihood space, as measured by the scaling behavior of KL divergence, are much smaller than in weight space, resulting in subdiffusive learning trajectories and early stabilization of language-model behavior despite weight drift.
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Publication details
- Journal / source
- Findings of the Association for Computational Linguistics: ACL 2026
- Publisher
- Association for Computational Linguistics
- Pages
- 23223–23248
- Publication date
- 1 January 2026
- Language
- EN