Optimizer Smoothing and the Grokking Phase Transition

Key findings: Under pure SGD with static weight decay, grokking on modular arithmetic exhibits a sharp phase transition: the grok rate drops from 83% to 0% within ΔWD = 0.001 (150 runs, 30 seeds per condition).Replications on modular multiplication (100 runs) and permutation composition on S₅ (300 runs, non-abelian) confirm the same cliff structure.Adding momentum suppresses grokking at all tested weight decays (450 runs, 30 seeds × 3 β values × 5 WD levels), consistent with the hypothesis that the race condition requires a memoryless optimizer.A Binder cumulant analysis across seven primes (735 runs) identifies task complexity p—not hidden width h—as the extensive variable, with WD_c ≈ 0.39/p (R² = 0.99).A scaling collapse yields a preliminary critical exponent ν ≈ 0.75 (variance 0.004).Under AdamW, the same structural reorganization is smoothed into a continuous crossover; a solution identity experiment confirms the two optimizers converge to genuinely different basins (CKA = 0.23).Transfer of spectral tools (participation ratio) back to SAT is clean but lacks predictive power (R² ≤ 0.01). A VSIDS decay sweep (420 solves) and decoupling experiment (980 solves) reveal that both SGD weight decay and VSIDS activity decay implement exponential forgetting competing with task-driven reinforcement.