Arcade: SLM Optimization via Orthogonal Decoupling of Latent State Spaces

We present Arcade-3B, a 3 B-parameter small language model (SLM) built onSmolLM3-3B with two complementary training innovations. First, we introduce SC-Orthogonal Training, a representation-space regularisation method grounded in theState–Constraint Orthogonality Hypothesis: the final hidden layer of a Transformer con-flates what information is stored (the State half S) with how it should be retrieved (theConstraint half C); imposing soft orthogonality between these two subspaces improvesgeneralisation without any architectural changes. Second, we introduce HashAdam, aCount-Sketch Adam variant that compresses optimizer states from O(N ) to O(K) perparameter via 2-Universal Hashing, reducing optimizer-state peak memory by ∼32×at negligible accuracy cost. On standard reasoning benchmarks Arcade-3B achieves62.9% on GSM8K (5-shot), 41.5% on HumanEval (pass@1), and 52.9% on MMLU (0-shot), surpassing comparably-sized public baselines while training on a single NVIDIAA100-80 GB GPU in under 24 hours. Code and weights are publicly available underApache 2.0.