Across generations, sizes, and types, large language models poorly report self-confidence in gastroenterology clinical reasoning tasks

npj gut and liver. 2026 Open access

Read full text
Authors: Nariman Naderi, Seyed Amir Ahmad Safavi‐Naini, Thomas Savage, Mohammad Amin Khalafi, Peter Lewis, Zahra Atf, Girish Nadkarni, Ali Soroush
DOI: 10.1038/s44355-026-00053-3
Cited by: 2
References: 15

Abstract

This study evaluated confidence calibration across 48 large language models (LLMs) using 300 gastroenterology board exam-style multiple-choice questions. Regardless of accuracy, all models demonstrated poor self-estimation of certainty. Even the best-calibrated systems (o1 preview, GPT-4o, Claude-3.5-Sonnet) exhibited substantial overconfidence (Brier scores 0.15–0.2, AUROC ~ 0.6). Models maintained high confidence regardless of question difficulty or response correctness. In their current form, LLMs cannot be relied upon to communicate uncertainty, and human oversight remains essential for safe use.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Clinical Reasoning and Diagnostic Skills Topic Modeling

Publication details

Journal / source
npj gut and liver.
Publisher
Springer Science and Business Media LLC
Volume
3
Issue
1
Publication date
5 February 2026
Language
EN