Accuracy and Knowledge Base Evaluation of ChatGPT-4o, Gemini-2.0-Flash, and DeepSeek-V3 in Metabolic and Bariatric Surgery: an Expert-Rated Blinded Study

Read full text Summarise with VIRAT
Authors: Mohamed Hany, Mohamed Hossam El-Din Zidan, Chetan D. Parmar, Shahab Shahabi Shahmiri, Hashem Altabbaa, Ahmed El Shamarka, Ahmed Amgad, Islam Abdelkhalek, Abdullah A. Assal, Marwan Emad Abdou, Mohammad Kermansaravi, MBS Global Expert Collaborative Group
+85 more Abdelrahman Ali Nimeri, Adel Karim Abou-Mrad, Ahmed Abokhozima, Ala Wafa, Amir Hossein Davarpanah Jazi, André Lázaro, Andrea Schroeder, Andrew G Robertson, Angelo Iossa, Anıl Ergin, Anna Casajoana, Anwar Ashraf Abouelnasr, Aparna Govil Bhasker, Ashraf J. Haddad, Asim Shabbir, Benjamin L. Clapp, Carlos Augusto Scussel Madalosso, Carlos Padrón, Cem Emir Güldoğan, Christine Karolina Stier, Cüneyt KIRKIL, Daniel Moritz Felsenreich, Richa Jaiswal, Ebrahim Aghajani, Estuardo J. Behrens, Farah A. Husain, Farnaz Rahimi, Ghulam Siddiq, Giovanni Lezoche, Heykel Mebarek, Hosam Mohamed Elghadban, Ivaylo Tzvetkov, Karl Peter Rheinwalt, Kazunori Kasama, Levon N. Grigoryan, Maria Antonieta Barrera, Mariano Palermo, Masoud Rezvani, Massimiliano Di Paola, Michael Talbot, Michel Gagner, Michel Vix, Miguel‐A. Carbajo, Mohamad Hayssam ElFawal, Mohamed Ibrahim Bahnasy, Mohamed Mokhtar Arafat, Mohamed Ammar, Mousa Khoursheed, Natán Zundel, Nikolaos B. Pararas, Nuru Bayramov, Otto Montoya, Panagiotis Laïnas, Paolo Gentileschi, Patrick Noe͏̈l, Paulina T. P. Salminen, Piotr Major, Ramen Goel, Rob Snoekx, Rodolfo Rodolfo, Rodrigue Chemaly, Rudolf Weiner, Rui José Silva Ribeiro, Ruth Blackham, Salvatore Tolone, Samer Gamil Mattar, Sara Gaafar Ibnauf Suliman, Sergio Carandina, Sergio O. Aparicio, Sergio Verboonen, Silvana Leanza, Silvia Leite, Sjaak Pouwels, Sonja Chiappetta, Stefano Olmi, Suhaib Ahmad, Tadeja Pintar, Tarek H. Hassab, Tigran Poghosyan, Tuna Bılecık, Valdemir José Alegre Salles, Vasileios Charalampakis, Wah Yang, Yannick Nijs, Yves Borbély
Cited by: 0
References: 30

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

BACKGROUND: Large language models (LLMs) are increasingly applied in medicine; however, their accuracy in guideline-driven, high-stakes specialties, such as metabolic and bariatric surgery (MBS), remains uncertain. This study evaluates the performance of ChatGPT-4o, Gemini 2.0 Flash, and DeepSeek-V3 in generating guideline-concordant responses to MBS clinical questions. METHODS: Thirty standardized, guideline-based MBS questions were presented to each model. Responses were randomized in order, anonymized (blinded as Model A/B/C), and evaluated by 93 MBS experts using a validated 0–3 scale (0 = inaccurate; 3 = fully guideline-concordant). A repeated-measures ANOVA with Bonferroni correction tested model differences; reliability was assessed with Cronbach’s α and intraclass correlation coefficients (ICC). RESULTS: DeepSeek-V3 achieved the highest mean score (2.44 ± 0.40), followed by ChatGPT-4o (1.79 ± 0.46) and Gemini 2.0 Flash (1.63 ± 0.47) (p < 0.001). Fully guideline-concordant ratings (score = 3) were most frequent for DeepSeek (80%) vs. ChatGPT (0%) and Gemini (3.3%). Internal consistency was excellent (α > 0.90), and inter-rater reliability was strong (ICC > 0.88). When mapped against the QUEST evaluation framework, the study addressed Quality and Understanding but did not fully capture Expression, Safety, or Trust dimensions. CONCLUSIONS: DeepSeek-V3 outperformed ChatGPT-4o and Gemini 2.0 Flash in generating guideline-concordant responses in MBS. These results highlight the need for ongoing, domain-focused validation before clinical use.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Artificial Intelligence in Healthcare Topic Modeling

Publication details

Journal / source
Obesity Surgery
Volume
36
Issue
5
Pages
2160–2171
Publication date
25 March 2026
Language
EN