SGD dynamics for anisotropic Gaussian mixtures

We study the dynamics of streaming SGD in the context of high-dimensional k-component Gaussian mixture models. We show that, when the data dimension d grows proportionally to the number of samples n, SGD converges to a deterministic equivalent, characterized by a system of ordinary differential equations. A key contribution of our technique is that it works for anisotropic data. As a simple example, I will discuss logistic regression on the 2-component model for various data covariance structures to illustrate the SGD dynamics for GMM with anisotropy. I will also discuss an extension of our methods to models with a growing number of components (k on order log(d)). Based on joint work with Inbar Seroussi.