Trained in the Light: Legibility as a Pretraining-Time Choice, Measured at Scale
Abstract
The alignment between a transformer's residual-stream writes and its unembedding readout is unconstrained by the prediction objective. In paired, seeded, from-scratch pretraining runs (162M and 70M, one architecture family), an unconstrained baseline's mid-layer writes drift to near-chance alignment with the readout span, while a twin trained with one additional loss term concentrates 44% of write energy into the top 8 of 768 readout directions (42x chance; 81% in the top 32), robust to a foreign lens. The price is small and positive on both seeds (matched-step 0.66-1.08%; paired bootstrap +1.15% CI [1.07,1.22]), rising to ~1.4% at 3.5x packing pressure. Reported at equal weight: the concentrated writes are no more token-decodable than baseline (visible is not readable), and unconstrained layers stay near chance. All predictions sealed and git-committed before launch; misses published at full weight; two adversarial review panels run pre-release and their findings incorporated. AI-collaboration disclosure included in the paper.
Fields of Study & Subjects
Advanced Data Storage Technologies
Semiconductor materials and devices
Parallel Computing and Optimization Techniques
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 20 July 2026
- Document type
- Preprint