Improving Radiology Report Error Detection Using a Multipass Large Language Model: Framework Development and Validation

Background: Large language model (LLM) proofreaders for radiology reports generate many false positives (FPs) due to the low prevalence of errors. Objective: This study aimed to determine whether an optimized LLM framework could improve both precision and cost-efficiency without compromising error detection capability. Methods: In this retrospective study, 1000 radiology reports (radiography, ultrasonography, computed tomography, and magnetic resonance imaging; 250 each) were sampled from the Medical Information Mart for Intensive Care III database. Two public chest radiography corpora (CheXpert and Open-i) served as external test sets. Three LLM frameworks were evaluated: single-prompt detector (framework 1); report extractor plus single-prompt detector (framework 2); and extractor, detector, and FP verifier (framework 3). Precision for each framework was assessed using positive predictive value (PPV) and detected errors per 1000 reports. Overall efficiency was estimated using model inference costs and reviewer labor costs. Results: PPV increased from 0.063 (95% CI 0.036-0.101) in framework 1 to 0.079 (95% CI 0.049-0.118) in framework 2 and 0.159 (95% CI 0.090-0.252) in framework 3 (P