Authors:
Mohamed Hany, Mohamed Hossam El-Din Zidan, Chetan D. Parmar, Shahab Shahabi Shahmiri, Hashem Altabbaa, Ahmed El Shamarka, Ahmed Amgad, Islam Abdelkhalek, Abdullah A. Assal, Marwan Emad Abdou, Mohammad Kermansaravi, MBS Global Expert Collaborative Group
+85 more
Abdelrahman Ali Nimeri, Adel Karim Abou-Mrad, Ahmed Abokhozima, Ala Wafa, Amir Hossein Davarpanah Jazi, André Lázaro, Andrea Schroeder, Andrew G Robertson, Angelo Iossa, Anıl Ergin, Anna Casajoana, Anwar Ashraf Abouelnasr, Aparna Govil Bhasker, Ashraf J. Haddad, Asim Shabbir, Benjamin L. Clapp, Carlos Augusto Scussel Madalosso, Carlos Padrón, Cem Emir Güldoğan, Christine Karolina Stier, Cüneyt KIRKIL, Daniel Moritz Felsenreich, Richa Jaiswal, Ebrahim Aghajani, Estuardo J. Behrens, Farah A. Husain, Farnaz Rahimi, Ghulam Siddiq, Giovanni Lezoche, Heykel Mebarek, Hosam Mohamed Elghadban, Ivaylo Tzvetkov, Karl Peter Rheinwalt, Kazunori Kasama, Levon N. Grigoryan, Maria Antonieta Barrera, Mariano Palermo, Masoud Rezvani, Massimiliano Di Paola, Michael Talbot, Michel Gagner, Michel Vix, Miguel‐A. Carbajo, Mohamad Hayssam ElFawal, Mohamed Ibrahim Bahnasy, Mohamed Mokhtar Arafat, Mohamed Ammar, Mousa Khoursheed, Natán Zundel, Nikolaos B. Pararas, Nuru Bayramov, Otto Montoya, Panagiotis Laïnas, Paolo Gentileschi, Patrick Noe͏̈l, Paulina T. P. Salminen, Piotr Major, Ramen Goel, Rob Snoekx, Rodolfo Rodolfo, Rodrigue Chemaly, Rudolf Weiner, Rui José Silva Ribeiro, Ruth Blackham, Salvatore Tolone, Samer Gamil Mattar, Sara Gaafar Ibnauf Suliman, Sergio Carandina, Sergio O. Aparicio, Sergio Verboonen, Silvana Leanza, Silvia Leite, Sjaak Pouwels, Sonja Chiappetta, Stefano Olmi, Suhaib Ahmad, Tadeja Pintar, Tarek H. Hassab, Tigran Poghosyan, Tuna Bılecık, Valdemir José Alegre Salles, Vasileios Charalampakis, Wah Yang, Yannick Nijs, Yves Borbély
Cited by:
0
References:
30
Abstract
BACKGROUND: Large language models (LLMs) are increasingly applied in medicine; however, their accuracy in guideline-driven, high-stakes specialties, such as metabolic and bariatric surgery (MBS), remains uncertain. This study evaluates the performance of ChatGPT-4o, Gemini 2.0 Flash, and DeepSeek-V3 in generating guideline-concordant responses to MBS clinical questions. METHODS: Thirty standardized, guideline-based MBS questions were presented to each model. Responses were randomized in order, anonymized (blinded as Model A/B/C), and evaluated by 93 MBS experts using a validated 0–3 scale (0 = inaccurate; 3 = fully guideline-concordant). A repeated-measures ANOVA with Bonferroni correction tested model differences; reliability was assessed with Cronbach’s α and intraclass correlation coefficients (ICC). RESULTS: DeepSeek-V3 achieved the highest mean score (2.44 ± 0.40), followed by ChatGPT-4o (1.79 ± 0.46) and Gemini 2.0 Flash (1.63 ± 0.47) (p < 0.001). Fully guideline-concordant ratings (score = 3) were most frequent for DeepSeek (80%) vs. ChatGPT (0%) and Gemini (3.3%). Internal consistency was excellent (α > 0.90), and inter-rater reliability was strong (ICC > 0.88). When mapped against the QUEST evaluation framework, the study addressed Quality and Understanding but did not fully capture Expression, Safety, or Trust dimensions. CONCLUSIONS: DeepSeek-V3 outperformed ChatGPT-4o and Gemini 2.0 Flash in generating guideline-concordant responses in MBS. These results highlight the need for ongoing, domain-focused validation before clinical use.