Systematic Mapping of LLM Knowledge Boundaries Across 67 Scientific Domains
Abstract
I map the knowledge boundaries of a 4-billion-parameter language model across 67 scientific domains using a log-probability oracle that compares P(truth) against P(best distractor) for 1,038 verified facts. The baseline model answers correctly on only 22.9% of facts. Failures concentrate in mathematical physics (NS regularity: mean margin −46.7), computational domains (0% in 15 domains), and facts involving specific quantitative relationships. Three systematic failure patterns account for most errors: token-length bias, frozen priors, and domain-specific wrong beliefs. Orthogonal adapter routing repairs all failures to 100%. The 1,038-fact verified dataset across 67 domains is released as an artifact.
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Mathematics
Computing
and Information Processing
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 16 March 2026
- Document type
- Preprint