Eeffctive Regularisation from Loss-Landscape Geometry: A Unied Derivation of Direction-Dependent Grokking Dynamics

Grokking is the phenomenon in which a neural network memorises its training data and then, after a prolonged delay, suddenly generalises [Power et al., 2022]. Two problems remain open: (1) grokking can occur even at weight-decay strength β = 0, and (2) a uniform penalty β||θ||^2 somehow produces direction-selective compression. We prove a main theorem that directly handles discrete SGD and non-quadratic loss surfaces. By analysing constrained minimisation of F(θ) = L_data(θ) + β||θ||^2 on the zero-loss manifold M_0 in the Hessian eigenbasis, we show that the effective decay rate γ_k in eigendirection v_k satisfies: [-log(1 - η(h_k + 2β)) / η] - C'_k * ε