Convergence Theory of Sharpness-Aware Minimization With Interpolating Neural Networks

View at publisher Summarise with VIRAT
Authors: Tan Sun, Yusu Hong, Junhong Lin
Cited by: 0
References: 39

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We present theoretical analysis of Sharpness-Aware Minimization (SAM) applied to training loss minimization with neural networks and smooth activations. Unlike prior works on stationary points measure in standard non-convex smooth optimization settings with noise assumptions, we leverage intrinsic properties of neural network loss landscapes to establish a convergence rate of Õ$(1/T )$for the average of training loss, under the smooth loss function and interpolating neural networks. Here,Tis the total number of iterations. Our analysis lies on the properties of self-bounded-like gradients and Hessians, distinguishing neural network optimization from general nonconvex smooth optimization. Our results require that the width$m \ge \tilde \Omega ( T )$for shallow neural networks while$m \ge \tilde \Omega (\sqrt T )$for deep neural networks (though with a minor different scaling architecture), both with a constant learning rate and a small perturbation parameter. The width requirement can be further reduced to$m ≥$Õ(1) for the logistic loss function. To our knowledge, our results are the first of its kinds for SAM in neural networks on training loss bounds.

Fields of Study & Subjects

Stochastic Gradient Optimization Techniques Neural Networks and Applications Advanced Neural Network Applications

Publication details

Journal / source
IEEE Transactions on Signal Processing
Volume
74
Pages
1491–1507
Publication date
1 January 2026
Language
EN