A grounded evaluation architecture for clinical scribes: a configuration-controlled study of LLM judges and a deterministic grounding floor
Abstract
Clinical scribe systems convert a spoken encounter into a structured note. Before that note is filed, an evaluation must flag what is unsafe for clinician review and, when it flags, produce a specific, typed, auditable reason. We evaluate LLM-as-judge configurations for this task and compare them with a deterministic grounding layer that verifies one property of a note against a defined source and returns a typed reason or declines. The study is a controlled ablation: one corpus of 54 by-construction cases and one grounding floor are held fixed, and the judge configuration is varied across five isolated workspaces, a single generalist sampled up to eight times, a six-model four-vendor ensemble, a same-model specialist council, a mixed-model specialist council, and a scalar reward baseline, each run twice, with and without the chart record in the judge's prompt, plus a calibration pass on two judges. The result is consistent across all of it, and the headline is about naming: a single judge names the injected upcode on 0 of 22 cases transcript-only and 6 of 22 with the record; only the six-model union ensemble reaches 22 of 22, and it pays 243 spurious flags for it. Verdict accuracy is the same to within one case (31 to 33 of 54) for every LLM configuration; every judge-side lever moves recall or self-consistency, and none lifts precision (the fraction of raised flags that are correct, the flag-level analogue of positive predictive value) out of a 0.09 to 0.19 band: fewer than one flag in five is right, under every configuration. Version 1 identified a corpus artifact that could have explained the band's absolute level and preregistered a corrected rerun with numeric decision rules (OSF 10.17605/OSF.IO/2ZU4H). This version reports that rerun (F11): with the artifact removed, the registered refutation condition fired in 8 of 8 LLM cells, every exonerating prediction failed, the within-run precision band is 0.076 to 0.149 and still configuration-invariant, so judge miscalibration, not the corpus, is the primary mechanism. Calibration tuning produces no held-out win on either judge and the accept-gate rejects it. On an external physician-curated arm, frontier judges under-detect, missing errors a clinician catches, and a preservation-and-attribution layer recovers each with a verbatim transcript span. Against all of this the floor is invariant within its scope: it false-cleared 0 of 22 upcodes (zero observed on 22 cases; the 95 percent upper bound is about 15 percent), actively confirmed 15 of 22 from a standard clinical terminology (SNOMED CT), and its validity checks flagged corpus cases whose codes do not exist or point at a different disease. The floor clears wrongful flags; it does not inject blocks: an upcode the judge never raises passes the gate untouched, which is why the two layers compose rather than compete. Stated bluntly, because the asymmetry is the deployment-critical fact: the floor can veto a wrong flag with cited evidence, but it cannot raise a flag the judge never raised, so a judge that approves an upcoded note leaves it approved, and the scalar baseline approving all 22 upcodes is exactly that scenario. [v2 erratum: version 1's abstract garbled the first sentence as "suppresses wrongful clears"; a wrongful clear is the event the floor never produces.] The judge is a screener no lever tunes into a decider; the floor is the only layer that returns a grounded, typed, auditable reason, and it is honest enough to audit the corpus that feeds it. We state the boundary of each claim in each section.Version 2 reports the preregistered corrected-corpus rerun (F11; OSF 10.17605/OSF.IO/2ZU4H): the registered refutation condition fired in 8 of 8 LLM cells, so judge miscalibration, not the corpus artifact, is credited as the primary mechanism. This record includes the full v2 reproducibility bundle: the scrubbed 44-case corpus, the deterministic scrubber and removal map, the per-case diff, the scored preregistration outcome, and per-case scorecards for all ten configuration-arm cells. See README.md.
Publication details
- Journal / source
- Zenodo (CERN European Organization for Nuclear Research)
- Publication date
- 13 July 2026
- Language
- EN