Statistical Physics of Deep Learning: Optimal Learning of a Multilayer Perceptron near Interpolation

For four decades, statistical physics has been providing a framework to analyze neural networks. A long-standing question remained on its capacity to tackle deep-learning models capturing rich feature-learning effects, thus going beyond the narrow networks or kernel methods analyzed until now. We positively answer through the study of the supervised learning of a multilayer perceptron. Importantly, (i) its width scales as the input dimension, making it more prone to feature learning than ultrawide networks and more expressive than narrow ones or ones with fixed embedding layers, and (ii) we focus on the challenging interpolation regime where the number of trainable parameters and data are comparable, which forces the model to adapt to the task. We consider the matched teacher-student setting. Therefore, we provide the fundamental limits of learning random deep neural-network targets and identify the sufficient statistics describing what is learned by an optimally trained network as the data budget increases. A rich phenomenology emerges with various learning transitions. With enough data, optimal performance is attained through the model’s “specialization” toward the target, but it can be hard to reach for training algorithms, which get attracted by suboptimal solutions predicted by the theory. Specialization occurs inhomogeneously across layers, propagating from shallow toward deep ones but also across neurons in each layer. Furthermore, deeper targets are harder to learn. Despite its simplicity, the Bayes-optimal setting provides insights into how the depth, nonlinearity, and finite (proportional) width influence neural networks in the feature-learning regime that are potentially relevant in much more general settings.