Data and Scripts for the paper "Signification Index"

Read full text Summarise with VIRAT
Authors: Xia
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We study whether different projection types within transformer neural networks develop systematically different levels of structural order during training. Using the Signification Index (S), a bounded metric in [0, 1] derived from the spectral entropy of weight matrix singular value decompositions, we analyze checkpoints from four language models spanning three architecture families, three training datasets, and two orders of magnitude in parameter count (160M to 70B parameters). We report two observational findings. First, at mature training stages (defined as tokens-per-parameter T/P ≥ 25), the MLP expansion projection consistently achieves a higher S than the contraction projection in all four models, regardless of whether this ordering holds from the start of training or emerges after a transient reversal. Second, in the two models with standard split attention projections (Amber-7B and OLMo-1B), the query projection achieves the highest S among query, key, and value projections at mature training stages. We discuss a plausible functional interpretation, i.e., projections serving more generative or query-generating roles may develop richer spectral geometry, while being explicit that the present study is observational, the sample of four models is small, and causal claims are not warranted. All data and analysis scripts are made available to support reproducibility.

Fields of Study & Subjects

Topic Modeling Explainable Artificial Intelligence (XAI) Neural Networks and Applications

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
27 March 2026
Language
EN