The Doodle Residual: Probing Language Model Knowledge Quality from Probability Distribution Geometry
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
We introduce the Doodle Residual, a measurement of language model knowledge quality derived from the geometric shape of token probability distributions during autoregressive generation. The participation ratio (effective number of competing tokens) has a strong linear relationship to top-1 probability. After subtracting this relationship, the residual correlates with topic-level factual accuracy across model scales from 3B to 405B parameters. On a 115-topic benchmark across 10 domains with automated accuracy labeling, the Doodle Residual achieves significance on 10 of 11 tested models spanning 5 architectures. Independent blind ground truth validation using two frontier LLM judges (Claude and GPT-4o) yields Cohen's d = 0.874 at 97.4% inter-judge agreement, confirming the measurement correlates with actual factual accuracy. The signal replicates across independent architectures with independent wall fits and independent accuracy labels, originates in pretraining, and is reshaped by alignment training. v1.1 (March 28, 2026): Corrected Qwen Coder 14B result (11/11 models significant), added independent blind ground truth validation (Cohen's d = 0.874), added HALT citation, enumerated all 17 features, minor consistency fixes.
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Adversarial Robustness in Machine Learning
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 28 March 2026
- Language
- EN
- Document type
- Preprint