Universality for high-dimensional stochastic gradient descent

A large family of high-dimensional statistical tasks share a common structure that their loss at a point in parameter space only depends on fixed-dimensional projections of the data (into the directions of the parameter and ground truth vectors). This includes mixture classification problems and single and multi-index models with one or two-layer networks. When the data distribution is isotropic Gaussian, and the parameter is trained using online stochastic gradient descent (SGD), certain low-dimensional families of “summary statistics” of the SGD trajectory evolve asymptotically autonomously, as the dimension and number of samples used go to infinity together. We consider the “universality” of these high-dimensional limits, namely whether or not they are sensitive to the details of product data distributions beyond their first two moments. Based on joint work with A. Jagannath.