Compositional Function Spaces for Deep Learning
Abstract
Abstract. We present a variational framework for studying functions learned by deep neural networks with rectified linear unit nonlinearities. We introduce a function space built from compositions of functions of second-order Radon-domain bounded variation. The compositional form of these functions captures the structure of deep neural networks. We prove a representer theorem that shows that deep neural networks with finite width solve regularized data-fitting problems over this space. The critical width is controlled by the square of the number of training data. This perspective explains the effect of weight-decay regularization in neural network training, the importance of skip connections, and the role of sparsity in neural networks. By considering the function-space perspective, we provide sharp links between deep learning and variational methods.
Fields of Study & Subjects
Geochemistry and Geologic Mapping
Machine Learning in Materials Science
Medical Imaging and Analysis
Publication details
- Journal / source
- SIAM Review
- Publisher
- Society for Industrial & Applied Mathematics (SIAM)
- Volume
- 68
- Issue
- 1
- Pages
- 127–149
- Publication date
- 9 February 2026
- Language
- EN