Comparative Evaluation of Large Language Models and Human Specialists in Acid-Base Disorder Interpretation and Sepsis Management: A Prospective Simulation Study (Preprint)

BACKGROUND Large language models (LLMs) such as ChatGPT and Google Gemini have demonstrated promising capabilities in medical reasoning and clinical decision support. However, their comparative performance against human specialists in critical care scenarios, particularly acid-base disorder interpretation and sepsis management, remains inadequately characterized. OBJECTIVE This study aimed to compare the diagnostic and therapeutic decision-making performance of advanced AI models (ChatGPT-4 and Google Gemini), a consensus-based ensemble AI approach, and human medical specialists in acid-base disorder interpretation and sepsis management scenarios using validated clinical vignettes. METHODS A total of 45 clinical case vignettes (20 acid-base disorder cases and 25 sepsis management cases) were developed by an expert panel. Cases were independently evaluated by 20 human specialists (10 emergency medicine physicians and 10 anesthesiologists), ChatGPT-4, Google Gemini, and a simple majority-voting ensemble model. Blinded evaluation was ensured throughout. Performance metrics included diagnostic accuracy, treatment recommendation appropriateness, and Surviving Sepsis Campaign (SSC) bundle compliance rates. RESULTS For acid-base disorder interpretation, the ensemble AI model achieved the highest overall accuracy (86.0%), followed by anesthesiologists (84.5%), ChatGPT-4 (83.7%), emergency physicians (83.2%), and Google Gemini (79.5%). In simple metabolic and respiratory disorders, AI models demonstrated comparable or superior performance to human experts (>90% accuracy). However, human specialists outperformed individual AI models in mixed acid-base disorders (humans: 75.5% vs ChatGPT: 68.5%, Gemini: 65.3%, P