Communication Before Computation: The Development of Communicative Structure in Transformer Learning
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Transformer language models coordinate computation across independently parameterized layers through a shared residual stream, yet the developmental process of this coordination remains poorly understood. We track the formation of inter-layer communicative structure across training in six models spanning four model families (Pythia, BLOOM, OLMo-2, TinyLlama). In the two families with dense early checkpoints (Pythia and OLMo-2), we observe a stereotyped developmental sequence: adjacent layers first align their dominant MLP transformation directions, establishing a shared communicative backbone; spectral restructuring then concentrates energy along these aligned directions, propagating as a backward cascade from the output layers. The mature form of the resulting communicative protocol depends on the activation function. GELU models (Pythia, BLOOM) consolidate weight-level alignment into a persistent, fixed-geometry communication channel. SwiGLU models (OLMo-2, TinyLlama) undergo sustained decoherence of static alignment, with coordination transferring to the learned gate, which dynamically selects shared effective subspaces as a function of the input. Preliminary analysis of the attention OV circuit reveals a similar pattern: weight-level alignment surges during backbone formation and then collapses, consistent with the attention pattern assuming the coordination role. This parallels the distinction between fixed and context-dependent communication subspaces documented across cortical circuits, suggesting that the static/dynamic dichotomy may be a general feature of coordination through shared subspaces in modular systems.
Fields of Study & Subjects
Topic Modeling
Explainable Artificial Intelligence (XAI)
Neural Networks and Applications
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 30 March 2026
- Language
- EN