Communication Before Computation: The Development of Communicative Structure in Transformer Learning

Read full text Summarise with VIRAT
Authors: Bray
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Transformer language models coordinate computation across independently parameterized layers through a shared residual stream, yet the developmental process of this coordination remains poorly understood. We track the formation of inter-layer communicative structure across training in six models spanning four model families (Pythia, BLOOM, OLMo-2, TinyLlama). In the two families with dense early checkpoints (Pythia and OLMo-2), we observe a stereotyped developmental sequence: adjacent layers first align their dominant MLP transformation directions, establishing a shared communicative backbone; spectral restructuring then concentrates energy along these aligned directions, propagating as a backward cascade from the output layers. The mature form of the resulting communicative protocol depends on the activation function. GELU models (Pythia, BLOOM) consolidate weight-level alignment into a persistent, fixed-geometry communication channel. SwiGLU models (OLMo-2, TinyLlama) undergo sustained decoherence of static alignment, with coordination transferring to the learned gate, which dynamically selects shared effective subspaces as a function of the input. Preliminary analysis of the attention OV circuit reveals a similar pattern: weight-level alignment surges during backbone formation and then collapses, consistent with the attention pattern assuming the coordination role. This parallels the distinction between fixed and context-dependent communication subspaces documented across cortical circuits, suggesting that the static/dynamic dichotomy may be a general feature of coordination through shared subspaces in modular systems.

Fields of Study & Subjects

Topic Modeling Explainable Artificial Intelligence (XAI) Neural Networks and Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
30 March 2026
Language
EN