Comparison of large language models and conventional machine learning in postoperative outcome prediction: a retrospective, multi-national development and validation study

BACKGROUND: Conventional machine learning (ML) models for predicting surgical outcomes have limitations in generalizability. We explored large language models (LLMs) as scalable alternatives to conventional ML models in predicting postoperative outcomes, including in-hospital 30-day mortality, intensive care unit (ICU) admission, and acute kidney injury (AKI). METHODS: This study utilized the Informative Surgical Patient for Innovative Research Environment (INSPIRE) dataset (n = 80 985) from South Korea for model development and internal validation, and the Medical Informatics Operating Room Vitals and Events Repository (MOVER) dataset (n = 6265) from the United States for external validation. The study compared three different LLMs-Generative Pre-trained Transformer [GPT]-4o, Llama-3-70B, and OpenBioLLM-70B-against MLs using various prompt engineering approaches. LLMs were evaluated with different model parameter quantizations (4-bit normalized floating point vs. 16-bit brain floating point). RESULTS: OpenBioLLM-70B was comparable to eXtreme Gradient Boosting (XGBoost) across all tasks (in-hospital 30-day mortality: area under receiver operating characteristic curve [AUROC] 0.782 [95% CI, 0.748-0.813] vs. 0.791 [95% CI, 0.753-0.825]; ICU admission: AUROC 0.595 [95% CI, 0.581-0.609] vs. 0.594 [95% CI, 0.580-0.608]; AKI: AUROC 0.830 [95% CI, 0.802-0.855] vs. 0.823 [95% CI, 0.792-0.851]) during external validation. Open-source LLMs maintained performance with 4-bit quantization, reducing computational requirements by 75%. CONCLUSIONS: The findings support the versatility and efficiency of LLMs for clinical decision support through on-premises compatibility, addressing data privacy. Further validation with diverse datasets is needed to ensure their reliability and applicability across different perioperative settings.