The Self-Knowledge Map: Where Language Models Do — and Do Not — Know What They Know

Read full text
Authors: Nathan Ryan Young
Cited by: 2
References: 1

Abstract

Does a language model know whether its own answer is correct? This paper measures the question as a MAP rather than a single number — across eight models from four families (0.5B-12B), two elicitation regimes, three self-report channels, and three internal "camera" channels, with pre-registered gates and bootstrap confidence intervals for every reported cell. Four findings organize the map. (1) An INTROSPECTION GAP — internal state predicting per-item correctness better than the model's own best self-report — is real but small and localized: on Qwen2.5-7B (zero-shot two-hop retrieval) a training-free internal camera reads correctness at 0.763 [0.691, 0.830] AUC versus the model's best self-report at 0.645 [0.562, 0.721], and uniquely in its family the gap persists under few-shot elicitation. (2) Self-knowledge is ELICITATION-GATED: the same self-report channel can move from chance to 0.90 AUC when the answer it judges is cleanly elicited (Gemma-2-2b-it P(True): 0.450 zero-shot -> 0.902 few-shot); much of what looks like missing self-knowledge is missing elicitation. (3) WHICH channel carries self-knowledge is jointly family- and regime-compiled — P(True) works on Qwen-3B and fails on Gemma in the same run where Gemma's answer log-probability reads 0.852; no channel is universal. (4) The receiving side has its own wall: handed verified readings of their own state — including ground truth — base models at every tested scale (0.5B-3B) revise indiscriminately in completion format (revision rate ~1.0 in every arm; the re-ask alone collapses accuracy) — the mirror requires an interface the model can hold, not just information it lacks. Two instrument failures are documented (a scorer that graded quiz-format non-answers as errors; an accent-blind string match), the corrected pipeline is published, and two of the authors' own interim claims are retracted at full weight. Companion verification deposit (every script, per-item result, the corrected pipeline, and CI artifacts): [DEPOSIT DOI]. Methods and drafting in close collaboration with Claude (Anthropic), disclosed throughout; every number traces to a sealed, pre-registered run.

Fields of Study & Subjects

Psychometric Methodologies and Testing Intelligent Tutoring Systems and Adaptive Learning Topic Modeling

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
19 July 2026
Document type
Preprint