Testing the spin-bath view of self-attention: A Hamiltonian analysis of GPT-2 transformer

The recently proposed physics-based framework by Huo and Johnson [arXiv:2504.04600]. models the attention mechanism of large language models as an interacting two-body spin system, offering a first-principles explanation for phenomena like repetition and bias. Building on this hypothesis, we extract the complete query-key weight matrices from a production-grade GPT-2 model and derive the corresponding effective Hamiltonian for every attention head. From these Hamiltonians we obtain analytic phase boundary and logit gap criteria that predict which token should dominate the next-token distribution for a given context. A systematic evaluation on 144 heads across 20 factual-recall prompts reveals a strong negative correlation between the theoretical logit gaps and the model's empirical token rankings (r≈-0.70, p