Order Capacity and Learnability in Deep Residual Networks: How Depth, Branch, and Normalization Jointly Determine Learnable Orders

Read full text Summarise with VIRAT
Authors: Chen Jiandong
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We study the maximum learnable Hermite order of deep networks and show that this question decomposes into two complementary sub-problems: expressiveness and trainability. For expressiveness, we identify two algebraic mechanismsa multiplicative order law governing composition and an additive order law governing pointwise productsthat jointly determine the order capacity of a network. Depth raises each block's capacity exponentially via composition, while a product-of-branches unit multiplies it proportionally to the number of branches. Combining both mechanisms yields a closed-form joint order capacity that scales as an exponential tower in depth and branch count. For trainability, we prove that residual connections provide a depth-independent identity gradient path whose structure exactly recovers the two-layer Order-Coupled Gradient Theorem, and that Pre-Norm(LayerNorm before each residual branch) ensures the required Gaussian conditions hold at every layer. Together, these two mechanisms guarantee that the order-resolved gradient norm remains bounded away from zero across all layers and all orders, so the algebraic order capacity is fully learnable. Furthermore, we prove an initial gradient signal factorization showing that multi-branch architectures replace the factorial gradient cost of a single-branch network with an exponential factor, providing a concrete trainability advantage. Six groups of numerical experiments on polynomial and non-polynomial activations conrm the theo-retical predictions.

Fields of Study & Subjects

Neural Networks and Applications Stochastic Gradient Optimization Techniques Advanced Neural Network Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
26 March 2026
Language
EN
Document type
Preprint