A Deterministic Validation Loop for LLM-Generated Scientific Code: Framework, Implementation, and Empirical Validation

Read full text Summarise with VIRAT
Authors: James Rice
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Abstract Large Language Models generate scientific code from training priors rather than user-defined specifications, silently substituting plausible but incorrect numerical coefficients---a failure mode we term specification drift. We present a five-component deterministic validation framework that detects and eliminates drift while preserving generative capacity: (1) Immutable Specification, (2) Prior-Drift Detection, (3) Adversarial Role Separation, (4) Statistical Validation, and (5) Persistent State with Termination Control. In a controlled experiment across two frontier LLMs (GPT-4o, Grok-3; $n = 10$ trials per model per coefficient), 95 of 96 blind measurements drifted from specification (Fisher's exact $p = 4.0 \times 10^{-10}$). Under the full framework, zero committed drift occurred across 7 checked parameters, validated over 120 simulation runs ($\sigma = 0.030$, CV $= 6.0\%$). The framework's statistical gating component independently caught a false positive before it could be published. We validate on SIMSIV (github.com/kepiCHelaSHen/SIMSIV), a 7,663-line agent-based simulation of human social evolution currently under review at JASSS. These results demonstrate that specification drift is a context problem amenable to deterministic engineering controls, with implications for any domain where LLMs generate calibrated scientific software.

Fields of Study & Subjects

Software Engineering Research Scientific Computing and Data Management Software Testing and Debugging Techniques

Publication details

Journal / source
Zenodo (CERN European Organization for Nuclear Research)
Publication date
25 March 2026
Language
EN
Document type
Preprint