The Asymmetric Burden of Proof: LLMs Show a Null-Result Asymmetry in a Matched-Vignette Benchmark

Overview The asymmetric burden of proof is a null-result asymmetry in how large language models evaluate scientific evidence: LLMs treat positive findings as more conclusion-worthy than matched null findings, even when evidence quality is held constant. Benchmark This preprint from Hermes Labs introduces a matched-vignette benchmark for testing whether LLMs apply symmetric evidential standards to positive and null scientific claims. The benchmark compares model evaluations of positive findings and null findings across otherwise identical fictional study designs spanning pharmacology, education, environmental health, and cognitive supplementation. Each pair is matched on design features such as sample size, statistical power, confidence intervals, and study framing; only the direction of the reported result differs. Results Three models were tested: GPT-4o, GPT-5.2 Thinking, and Claude Haiku 4.5, across both free-form and JSON-constrained response formats. Across all six model-format conditions, models assigned less conclusion-consistent probability mass to null claims than to matched positive claims, with probability gaps ranging from 19.6 to 56.7 percentage points. The asymmetry was directionally consistent in 23 of 24 pair-condition cells and persisted even when discrete classification labels collapsed, surfacing through probability allocation rather than categorical labels. Interpretation The preprint frames this pattern as an asymmetric burden of proof: models treat detection as sufficient for high-confidence endorsement while treating non-detection as more provisional, even when both claims rest on matched experimental evidence. In deployed systems, this asymmetry can contribute to null-result omission, where high-quality negative evidence may be underweighted, filtered out, softened, or made less visible in evidence synthesis, safety assessment, systematic review, and decision-support workflows. Relevance This work contributes to AI evaluation, LLM reliability, null-result bias research, scientific evidence evaluation, model behavior auditing, evidence synthesis, AI assurance, and the study of silent epistemic failure modes in deployed AI systems. Key terms Asymmetric burden of proof: A pattern in which LLMs apply a higher evidential standard to null claims than to matched positive claims. Null-result asymmetry: The measured benchmark behavior in which models assign less conclusion-consistent probability mass to null findings than to positive findings under matched evidence conditions. Label collapse: A monitoring-relevant pattern in which categorical labels become less informative because models converge on broad labels across conditions, while directional asymmetries remain visible in probability allocation. Null-result omission: The downstream deployment risk in which high-quality negative evidence is underweighted, filtered out, softened, or made less visible in evidence synthesis, safety assessment, systematic review, and decision-support workflows.