Conformal Scaling in Trained Transformer Attention: Evidence for an SYK Fixed Point

Read full text Summarise with VIRAT
Authors: Ariel Umphrey, Eldon Umphrey
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We report the first empirical measurement of conformal scaling in trained transformer attention. Analyzing all 144 attention heads in GPT-2 with maximum-entropy inputs, we find that 44 heads exhibit power-law attention weight decay with R² > 0.90 and median conformal dimension Δ = 0.2493, matching the Sachdev-Ye-Kitaev (SYK) model prediction Δ = 1/4 for q = 4 interactions in one dimension. Three controls isolate contributions: reinitializing attention weights destroys all power-law structure; randomizing positional embeddings preserves the power law but shifts Δ; structured text inputs shift the effective Δ. GPT-2 and Pythia — architecturally distinct positional encoding schemes — both converge toward Δ = 1/4 with model depth. Training dynamics across three Pythia models reveal a phase transition from disordered to conformal attention. The block entanglement entropy follows S(k) = (c/3)log(k) with R² > 0.99, the entanglement structure predicted by conformal field theory. Fine checkpoint sampling reveals a prethermal plateau at Δ ≈ 0.50 (the SYK q=2 integrable value), suggesting a two-stage flow from disorder through integrable order toward chaos.

Fields of Study & Subjects

Quantum many-body systems Topic Modeling Theoretical and Computational Physics

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
25 March 2026
Language
EN
Document type
Preprint