Information-Geometric Degeneration in Large Language Models: Fisher Metric Collapse, Mixture Singularities, and Modular Specialization

This paper develops an information-geometric framework for analyzing the internal structure of large language models (LLMs). The study shows that several widely observed phenomena in modern LLMs—such as covariance cutting between modules, modular specialization in attention and Mixture-of-Experts (MoE) architectures, effective rank collapse, and the curvature of empirical scaling laws—can be understood as consequences of two fundamental geometric processes: Fisher metric degeneration and mixture-model singularity. The Fisher information matrix of an LLM becomes increasingly block-structured as cross-module interactions weaken, leading to a loss of effective Fisher rank. This degeneration reflects a reduction in the number of information-geometrically active directions, providing a geometric explanation for dimensional collapse in the model’s representation space. The paper formalizes this behavior using the eigenvalue spectrum of the Fisher metric and an entropy-based notion of effective Fisher rank. In architectures with modular specialization, such as MoE layers or highly specialized attention heads, the statistical manifold induced by the model develops singular regions analogous to those found in classical mixture models. These singularities arise from non-identifiability and redundancy, and their severity increases as gating distributions become sharper. The paper clarifies how these mixture singularities correspond to topological redundancy in the companion work on stratified topology. The study further interprets the curvature of empirical scaling laws as a manifestation of entropy saturation in the Fisher spectrum: beyond a certain scale, additional parameters do not significantly increase the manifold’s effective dimension. This provides a structural explanation for diminishing returns in model scaling. Overall, the paper offers a unified information-geometric account of connectivity loss, specialization, and dimensional degeneration in LLMs, complementing topological and physical perspectives and providing a foundation for future theoretical and architectural research.