Opening Internal Memory in Language Models
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Dominant hidden-state language-model architectures do not simultaneously make persistent state inspectable, exactly intervenable, optimization-theoretically analyzable, and performance-predictable before training.This paper studies an alternative normal form: past spans become named persistent objects inside the model, and every memory-mediated influence reaches the token prediction through one explicit read/write interface.That single design constraint yields a wider theorem stack than interpretability alone.On the representation side, predictive state, separator objectification, exact intervention semantics, and predictively safe close and write operations motivate addressable persistent objects.On the optimization side, the retrieval mean map is a global diffeomorphism onto the memory-hull interior; single-token and untied multi-token read objectives are benign; shared read heads reduce sharply to linear approximation of the untied optimal query field; decoder and fixed-route write blocks are convex; and tokenwise joint read/write has no spurious finite local minima.On the predictability side, off, uniform, oracle, and fit-residual diagnostics define a certified pre-training performance band and a routing-margin test.On the scaling side, the read-side complexity bound depends on geometry constants but not on memory-bank size or frozen-backbone parameter count, and the targeted gains are orthogonal to simply enlarging the native context window.Empirically, a frozen Qwen3-8B backbone with a trainable memory pathway reaches tail NLL $1.829$ on 5,000 held-out examples, versus 1.896 for uniform retrieval and 1.899 for retrieval-off, and exact span-level interventions show that retrieved objects matter causally.The paper therefore reframes explicit memory not only as an interpretability device, but as a trainable, diagnosable, and pre-analyzable memory substrate, while reserving fixed experimental slots for the longer-horizon utility/window-bypass and structural-assumption-validation programs that the theory makes especially clean.
Fields of Study & Subjects
Explainable Artificial Intelligence (XAI)
Topic Modeling
Natural Language Processing Techniques
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 2 April 2026
- Language
- EN
- Document type
- Preprint