Comparisons between Clinician and AI Predictions of Patient Outcomes in the Intensive Care Unit

Accurate prediction of patient outcomes in the intensive care unit (ICU) is essential for improving patient care, optimizing resource allocation, and supporting clinical decision-making. Although machine learning (ML)-based artificial intelligence (AI) models have demonstrated strong predictive performance in retrospective studies, direct comparisons with human clinicians, particularly in prospective, real-world settings, remain limited. This study addressed this gap by comparing the predictive performance of ML models and clinicians for multiple ICU outcomes in both retrospective and prospective settings. Retrospective clinician predictions were obtained for 990 ICU admissions from Alberta’s 15 adult tertiary ICUs, and prospective clinician predictions were collected at the bedside for 238 ICU admissions between 16 and 24 hours after ICU admission. ML models were trained on data from 46,631 ICU admissions (41,096 unique patients) and evaluated on the same cases assessed by clinicians. Predicted outcomes included in-hospital mortality, 30-day post-discharge mortality, ICU and hospital length of stay, delirium, and acute kidney injury. ML models generally outperformed individual clinicians across outcomes in the retrospective setting, although clinician performance sometimes exceeded ML when predictions were aggregated. In the prospective setting, subspecialized physicians tended to match or exceed ML performance, whereas trainees and nurses generally performed worse than AI. Prediction of the length of stay outcomes were poor for both clinicians and ML, and inter-rater agreement was consistently low to fair both among clinicians and between clinicians and ML. These findings provide a comprehensive benchmark for clinician and AI performance in ICU prognostication and clarify the distinct strengths of human and ML-based prediction. Together, these findings underscore the need for prospective evaluation when assessing clinical AI tools and highlight the complementary roles that clinician expertise and data-driven models may play in critical care.