Lyapunov Exponents for Attention Composition: A Dynamical Systems Perspective on Deep Transformers

I develop the first Lyapunov exponent framework for analyzing eigenvalue dynamics in composed attention layers, bridging transformer theory with dynamical systems. Key contributions: (1) First computation of the full Lyapunov spectrum for attention products, proving Λ₁ = 0 exactly and Λₖ < 0 for k > 1 (2) Quantification of temperature effects on spectral collapse rates (3) Refined closed-form formula for predicting rank collapse depth (4) Discovery that non-commutative attention matrices exhibit Lyapunov structure distinct from naive eigenvalue products (5) Precise quantification of how residual connections reduce contraction rates by 2.4× All theoretical results are experimentally verified. Code available at: https://github.com/Tylerbryy/lyapunov-attention