The Geometry of Learning: Scale-Dependent Rotation Fragility in Transformers

We report a scale-dependent geometric phenomenon in transformer language models: sensitivity to rotational perturbations in parameter space decreases dramatically with model size. Key findings:1. The phenomenon is non-trivial: Random (untrained) networks show R ≈ 0% at all scales. Only trained networks exhibit scale-dependent fragility.2. Small trained models are rotation-fragile: R = +38.6% [95% CI: 35.5–41.7] at 160M parameters.3. Large trained models are rotation-tolerant: R = +1.9% [95% CI: −0.1–3.9] at 1B parameters.4. The pattern is epsilon-invariant: Small models show 10–30× higher rotation fragility across all tested perturbation magnitudes.5. The pattern holds across architectures: Both Learned PE and RoPE architectures follow qualitatively similar scaling. Our null-model control provides strong evidence against the trivial explanation that "larger models absorb more noise." This introduces rotation fragility as a measurable signature of representational geometry with implications for robustness and interpretability research. Code available at: https://github.com/atomic-trust/velado-transition