Data and Scripts for the paper "Signification Index"
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
We study whether different projection types within transformer neural networks develop systematically different levels of structural order during training. Using the Signification Index (S), a bounded metric in [0, 1] derived from the spectral entropy of weight matrix singular value decompositions, we analyze checkpoints from four language models spanning three architecture families, three training datasets, and two orders of magnitude in parameter count (160M to 70B parameters). We report two observational findings. First, at mature training stages (defined as tokens-per-parameter T/P ≥ 25), the MLP expansion projection consistently achieves a higher S than the contraction projection in all four models, regardless of whether this ordering holds from the start of training or emerges after a transient reversal. Second, in the two models with standard split attention projections (Amber-7B and OLMo-1B), the query projection achieves the highest S among query, key, and value projections at mature training stages. We discuss a plausible functional interpretation, i.e., projections serving more generative or query-generating roles may develop richer spectral geometry, while being explicit that the present study is observational, the sample of four models is small, and causal claims are not warranted. All data and analysis scripts are made available to support reproducibility.
Fields of Study & Subjects
Topic Modeling
Explainable Artificial Intelligence (XAI)
Neural Networks and Applications
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 27 March 2026
- Language
- EN