Latent Algorithmic Structure Precedes Grokking: A Mechanistic Study of ReLU MLPs on Modular Arithmetic

Read full text Summarise with VIRAT
Authors: Anand Swaroop
Cited by: 0
References: 2

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Grokking on modular addition has been characterized by sinusoidal representations in transformers and MLPs. We find that ReLU MLPs in our setting instead learn near-binary square-wave input weights, with intermediate values appearing only near sign-change boundaries, and output weights whose dominant Fourier phases satisfy $ϕ_{\mathrm{out}} \approx ϕ_a + ϕ_b$. These relations persist under label noise and even in models that do not grok. Using DFT-extracted frequencies and phases to replace learned weights with ideal square/cosine waves yields 95.8% accuracy from noisy models that achieve only 0.40%. This suggests that grokking sharpens an algorithm already encoded during memorization.

Fields of Study & Subjects

Neural Networks and Applications Domain Adaptation and Few-Shot Learning Advanced Neural Network Applications

Publication details

Journal / source
arXiv (Cornell University)
Publication date
20 March 2026
Language
EN
Document type
Preprint