Abstract
Can AI coding agents understand not only the syntax of Earth System Model (ESM) code but also the physics it encodes? Recent benchmark work has substantially im- proved evaluation for software engineering, scientific programming, and scientific agents, but those settings rarely test repository-level edits to production ESM code under explicit physical-validity constraints. Climate science, by contrast, demands more than executable syntax: a code change can look plausible as software while still targeting the wrong routine, implementing the wrong physical mechanism, or introducing locally non-physical logic. We introduce ESM-BENCH, a benchmark that directly probes this physics-and- code understanding gap. ESM-BENCH currently contains 107 tasks and covers 3 open-source ESM repositories, including Noah-MP, CLM5, SUMMA, ParFlow, MOM6, VIC, WRF, and E3SM. Together these repositories represent Fortran 90, Fortran 2008, and C across land-surface, ocean, hydrology, atmosphere, and coupled Earth-system domains. Tasks span 4 scientifically grounded categories: physics-based bug fixes (PBF, n= 39), process representation modifications (PRM, n= 47), parameterization scheme selection (PSS, n= 15), and parameter opti- mization (PO, n= 6). Each task provides (1) a physics-driven problem description; (2) the full source file contents at the parent commit; (3) the ground-truth unified diff from the real developer commit; and (4) review metadata for physics-aware human evaluation, including a six-item rubric. The benchmark is organized as two tracks: Track A tests repository localization (can the agent find the right file?), while Track B tests physics-aware patch synthesis at three oracle-derived hint levels, namely baseline (description + source files), oracle-localized (adds subroutine and line hints), and expert-assisted (adds edit context + Fortran idioms), enabling controlled measurement of whether the bottleneck is code navigation, code generation, or domain knowledge. The central design principle is that physics-aware patch review and exact-match recovery are independent dimensions: an agent can recover the rough intent of a fix without reproducing the exact developer diff, and it can emit a syntactically valid diff while still making a locally non-physical change. ESM-BENCH measures both. Every task carries explicit commit-time provenance metadata, enabling model-specific temporal filtering and contamination audits.