A Theoretical Framework for Ordered Fisher Metric Degeneration: Foundations and Application to Grokking

Read full text Summarise with VIRAT
Authors: Yusuke Tanaka
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We propose a theoretical framework in which constraint accumulation drives ordered degeneration of Fisher metric across network layers. As the constraint parameter κ = λt increases with training, the information content of each layer degenerates exponentially at a layer-specific rate α_l under a decay-dominated regime, reaching a collapse point κ_c^l = (1/α_l) log(I_l(0)/ε). We derive under simplifying assumptions, via an input signal propagation model, that α_l = 2(l−1)η: output-proximal layers accumulate constraint faster because the input signal reaching them has passed through more weight-decay-affected layers. This provides a coherent structural account consistent with all three phases of grokking — memorization, generalization, and anti-grokking collapse. The framework's potential applicability to a broader class of constrained learning phenomena is identified as a direction for future investigation under the universality hypothesis. The present paper provides a self-contained treatment; the framework was originally motivated by Unified Constraint Dynamics (UCD), a broader theoretical program for ordered degeneration under finite-resource constraints (Tanaka, 2026, Zenodo DOI: 10.5281/zenodo.18828954). v2: Added Transformer experiment (Experiment 2), ordered collapse timing table (Table 1), Figure 2 with grokking at step 6,250. Theorem numbering unified (Theorem 1).

Fields of Study & Subjects

Neural Networks and Applications Stochastic Gradient Optimization Techniques Domain Adaptation and Few-Shot Learning

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
22 March 2026
Language
EN
Document type
Preprint