From Collapse to Chaos: Metastable Dynamics in Inertial Attention Mechanisms

Standard Transformer architectures model token mixing as a sequence of discrete, overdamped updates, mathematically isomorphic to gradient descent on a static energy landscape. We demonstrate that this formulation inherently suffers from "Rank Collapse" in deep layers, as trajectories inevitably converge to local fixed points, leading to a loss of representational diversity. This paper reformulates the attention mechanism as a Second-Order Inertial Neural ODE. By defining a Hamiltonian system and introducing momentum via Symplectic Verlet Integration, we enable the hidden state to escape the gravitational pull of individual key vectors. The system enters a regime of Metastable Itinerancy (or Transient Chaos), characterized by a near-zero Lyapunov exponent. Furthermore, we illustrate how this inertial formulation unifies Transformers with State Space Models (SSMs), providing a rigorous physical basis for infinite-context signal propagation without computational overhead.