Grokking Has Finite Capacity: Measuring and Overcoming Limits on Simultaneous Algorithmic Discovery

Read full text Summarise with VIRAT
Authors: John Kearney
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Neural networks have a finite capacity for algorithmic discovery through grokking, the phenomenon where models generalize long after memorizing training data. Using modular arithmetic as a testbed, we show that a d=128 transformer reliably groks up to 5 simultaneous operations but collapses completely at 6. This collapse is not gradual: across all 5 random seeds, the model fails on every operation, including ones it trivially solves alone. We localize the interference to the shared embedding layer and show two interventions that recover full performance: separate per-operation embeddings (a minimal fix at this scale) and a split architecture with independent modules (a general solution). An automatic pipeline discovers optimal operation groupings through gradient similarity analysis with no domain expertise. The split achieves equivalent performance at roughly half the parameters of the smallest monolithic model that can match it. These results suggest that as models are asked to learn more diverse algorithms, some form of representational separation will be necessary to avoid capacity limits.

Fields of Study & Subjects

Neural Networks and Applications Domain Adaptation and Few-Shot Learning Topic Modeling

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
31 March 2026
Language
EN
Document type
Preprint