The Geometric Origin of Logic: Weight Polarization and Manifold Collapse in Transformers

How large language models acquire structured behaviors—such as arithmetic consistency—from unstructured data remains poorly understood. In this work, we present empirical evidence that logical capabilities may emerge not merely from scale or data, but from geometric constraints imposed during training. Using a Vector-Gravity Transformer (VGT) with an L² norm constraint on hidden states, we observe that a small model trained for only 20,000 steps achieves 70.8% exact-match accuracy on six-digit addition, while a standard Transformer baseline scores 0.0%. Further analysis reveals a striking phenomenon: the output layer weights undergo significant polarization (standard deviation increases by 126%), and localized high-magnitude activations appear in the weight matrix. These findings support a hypothesis: logic is not “learned” through memorization, but is “squeezed out” under geometric pressure, forcing the network to construct algorithmic circuits within a compressed representational manifold.