Comparative Analysis of ChatGPT-4o and OpenEvidence for Decision Support in Hybrid Coronary Revascularization

Background This study assessed whether the Large Language Models (LLMs) ChatGPT-4o and OpenEvidence could generate accurate recommendations for complex coronary revascularization using expert consensus. Methods Forty clinical scenarios were constructed using the European Society of Cardiology Working Group on Cardiovascular Surgery and European Association of Percutaneous Cardiovascular Interventions position paper on Hybrid Coronary Revascularization (HCR). LLMs were presented with clinical scenarios and prompted to generate a clinical management recommendation. Responses were scored by three independent raters on a 5-point Likert scale (5 = completely correct, 1 = completely incorrect). Primary outcomes were median score, interquartile range (IQR), proportion of perfect responses (score = 5), and proportion of acceptable responses (score ≥ 4). Paired ordinal data was compared using the Wilcoxon signed-rank test. Paired categorical outcomes were analyzed using McNemar's test. Results Interrater reliability was good (ICC(2,1) = 0.83; 95% CI, 0.76-0.88). The median score was 5.0 (IQR 4.0–5.0) for ChatGPT-4o and 4.0 (IQR 4.0–5.0) for OpenEvidence. ChatGPT-4o achieved significantly more perfect responses (72.5% vs 45%; McNemar p = 0.013). Acceptable responses occurred in 92.5% of ChatGPT-4o responses and 100% of OpenEvidence responses (p = 0.25). Three ChatGPT-4o responses were categorized as unacceptable (score ≤ 3), whereas no unacceptable responses were observed for OpenEvidence. Conclusion Both LLMs demonstrated high performance in clinical decision-making for HCR. While ChatGPT-4o achieved more perfect responses , OpenEvidence trended towards more consistently acceptable responses . Findings highlight potential for both general-purpose and domain-specific LLMs in supporting evidence-based cardiovascular care.