Systematic Mapping of LLM Knowledge Boundaries Across 67 Scientific Domains

Read full text
Authors: Bryan Sanchez
Cited by: 6

Abstract

I map the knowledge boundaries of a 4-billion-parameter language model across 67 scientific domains using a log-probability oracle that compares P(truth) against P(best distractor) for 1,038 verified facts. The baseline model answers correctly on only 22.9% of facts. Failures concentrate in mathematical physics (NS regularity: mean margin −46.7), computational domains (0% in 15 domains), and facts involving specific quantitative relationships. Three systematic failure patterns account for most errors: token-length bias, frozen priors, and domain-specific wrong beliefs. Orthogonal adapter routing repairs all failures to 100%. The 1,038-fact verified dataset across 67 domains is released as an artifact.

Fields of Study & Subjects

Topic Modeling Natural Language Processing Techniques Mathematics Computing and Information Processing

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
16 March 2026
Document type
Preprint