Bridging the gap between AI and clinicians: an exploratory study on sepsis prognosis comparing ChatGPT-4o, DeepSeek-R1, and expert physicians

Background Sepsis remains a leading cause of ICU mortality, with diagnostic, therapeutic, and prognostic challenges. Large language models like ChatGPT and DeepSeek show potential in clinical decision-making but lack evaluation in sepsis management. Methods This exploratory study enrolled 52 sepsis patients admitted to three ICUs of different levels in China between December 2024 and February 2025. A double-blind design was used, with AI-generated diagnostic and therapeutic plans standardized and anonymized. Clinical experts independently scored outputs across six dimensions: diagnostic accuracy, therapeutic rationality, monitoring feasibility, treatment-effect prediction, overall adaptability, and prognostic accuracy. Cross-evaluation was also performed. Analyses included Wilcoxon signed-rank tests, effect sizes, intraclass correlation coefficients (ICC), receiver operating characteristic curves, Brier scores, calibration, Bland–Altman plots, and subgroup analyses. Results The cohort had a mean age of 60.9 ± 11.0 years, and 76.9% were female. Complication rates were 50.0% for ARDS, 40.4% for AKI, and 38.5% for multiple organ dysfunction. Diagnostic accuracy was high for both models (94.2% ChatGPT-4o; 96.2% DeepSeek-R1), exceeding physician performance in complication recognition. Cross-evaluation showed ChatGPT-4o consistently rated DeepSeek-R1 higher, with significant differences in therapeutic rationality, monitoring feasibility, and prognostic accuracy (P < 0.001). Inter-rater reliability was good (ICC = 0.63). Outcome validation demonstrated best performance in AKI prediction (AUC 0.85 vs. 0.80), with DeepSeek-R1 superior in 28-day mortality prediction, though both models performed poorly in ARDS prediction. Conclusion ChatGPT-4o and DeepSeek-R1 demonstrated high diagnostic accuracy and complementary strengths, but limitations remain in individualized therapy and ARDS prediction. Future work should focus on real-time AI–clinician collaboration models integrating dynamic data.