Destructive Rank Collapse: The Unused Capacity Paradox

Residual connections are ubiquitous in modern deep learning, from ResNets and Vision Transformers to language and diffusion models, yet their necessity has remained poorly understood. We identify Destructive Rank Collapse (DRC), a geometric failure mode that explains why identity-preserving pathways are essential in deep networks. Across 15,315 training runs spanning MLPs, CNNs, and Vision Transformers on synthetic and real datasets, we show that removing residual connections causes hidden representations to collapse early to effectively rank-1 subspaces, destroying class-discriminative structure before learning can succeed. Crucially, normalization mechanisms (BatchNorm, LayerNorm) stabilize gradient magnitudes but do not prevent rank collapse, revealing a clear separation between scalar stability and geometric stability. Using random matrix theory, we show that depth induces exponential singular value separation in feedforward networks, while residual connections impose a geometric floor that preserves representational dimensionality, explaining the Unused Capacity Paradox. These effects persist from few-shot to full-data regimes, establishing DRC as a depth-driven phenomenon and clarifying why successful deep learning requires both normalization and residual connections.