Uncertainty-aware extraction of clinical findings from Finnish EHRs using open large language models

Abstract Objective To evaluate whether open-weight large language models (LLMs) can accurately extract clinical findings from Finnish-language pediatric records, and whether prediction uncertainty can be used to triage cases for expert review to minimize manual work Materials and Methods Retrospective cohort of 97 pediatric ischaemic stroke patients (1 month–17 years) from Helsinki University Hospital (2010–2023). Three open LLMs (gpt-oss-20b, DeepSeek-R1-Distill-Qwen-32B, and medgemma-27b-text-it) were prompted in English to detect four extraction targets (hemiplegia, headache, seizure, and stroke as a positive control) from each patient’s full free-text record. Each combination received 15 calls (five temperatures × three repeats). Performance was benchmarked against a clinician reference (accuracy, recall, precision, F1). Shannon entropy across the 15 calls quantified within-model uncertainty; inter-model disagreement provided an ensemble signal. Patients were ranked by uncertainty for a simulated selective-review workflow. Findings were externally validated in an independent neonatal stroke cohort (n = 88). Results Gpt-oss-20b achieved the best balance of recall (0.91–1.00) and precision (0.83–0.92), with F1 0.89–0.95 across non-control extraction targets. Entropy in misclassified cases was 2.4–3.4 times higher than in correctly classified cases. Entropy-based triage achieved complete error coverage by reviewing