Implicit regularization of normalized gradient descent
Abstract
How to find flat minima? We propose running normalized gradient descent, usually reserved for nonsmooth optimization, with sufficiently slowly diminishing step sizes. This induces implicit regularization towards flat minima if an appropriate Lyapunov functions exists in the gradient dynamics. Our analysis shows that implicit regularization is intrinsically a question of nonsmooth analysis, for which we deploy the full power of variational analysis and stratification theory.
Fields of Study & Subjects
Stochastic Gradient Optimization Techniques
Optimization and Variational Analysis
Advanced Optimization Algorithms Research
Publication details
- Journal / source
- arXiv (Cornell University)
- Publication date
- 9 February 2026
- Language
- EN
- Document type
- Preprint