The residual growth landscape: A 43-model survey of residual stream dynamics and a post-hoc intervention study

Read full text Summarise with VIRAT
Authors: Katsuki Ono
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We measure the residual stream growth factor, the ratio of the last-layer residual norm to the first-layer norm, across 43 openly available language models spanning 15 architecture families and 70M to 4B parameters. Residual growth varies by over 500x across models (from 5x in OLMo-2 to 2,747x in Qwen3-1.7B) and shows no correlation with parameter count (Spearman r = 0.043, p = 0.78). We conduct two intervention studies on 9–10 models. First, Norm Equalization (NormEq), an analytical rescaling that forces uniform residual growth, degrades perplexity in 8 of 9 cases, with catastrophic failure (+2,073%) in Qwen3-0.6B. Second, progressive layer dropping reveals that resilience to depth reduction is uncorrelated with residual growth (r = -0.09, p = 0.80): Falcon-H1 (RG = 9x) is the most fragile model, while GPT2-XL (RG = 510x) degrades gracefully. We conclude that heterogeneous residual growth is a learned feature of Pre-LN Transformer training, not an architectural defect, and that layer-level criticality depends on architecture type rather than residual dynamics. Residual growth values reported in the survey and intervention files were measured under different calibration and evaluation setups, so overlapping models may have different absolute RG values across files. Cross-file comparisons should therefore use within-experiment values.

Fields of Study & Subjects

Topic Modeling Advanced Neural Network Applications Natural Language Processing Techniques

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
28 March 2026
Language
EN