T³: Toroidal Tesseract Transformer - Geometric Optimization for Deep Learning

Read full text
Authors: Garret Sutherland
Cited by: 0

Abstract

We present T3 (Toroidal Tesseract Transformer), a framework that respects the geomet- ric structure of neural network weight space. Standard optimizers treat weight space as Euclidean, but we demonstrate that emergent structure follows geometric invariants, and respecting this geometry yields significant improvements. We introduce three components: a geodesic optimizer that navigates a 3-torus man- ifold (32–38% perplexity improvement, p < 0.001), a chain architecture that preserves geometry at scale (7–33% improvement, p < 0.0001), and adaptive attention via E,I,F,V primitives. We validate our approach across 75 experimental conditions, showing that the nearest-neighbor (NN) invariant holds at NN ≈ 1.0 (CV = 12.5%). This invariant caught a critical +0.1 bug that was degrading performance by 30%. We report both successes and the training recipe that enables them: primitives require staged training (freeze base → train heads → fine-tune). With staged training, primitives achieve +22–30% perplexity improvement (p < 0.02) on WikiText and C4. End-to- end training from scratch fails catastrophically. The same mechanism yields opposite results depending on training procedure

Fields of Study & Subjects

Topic Modeling Stochastic Gradient Optimization Techniques Neural Networks and Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
31 January 2026
Document type
Preprint