Comparing ChatGPT-4 and Google Gemini’s Ability to Accurately Answer and Explain General Surgery Residency American Board of Surgery In-Service Training Examination (ABSITE) Practice Questions: Comparative Observational Study (Preprint)

BACKGROUND There is an increasing interest in the role of Large Language Models (LLM) in medical education. LLMs as resources to study for licensure examinations have not been validated. OBJECTIVE As an initial step to understand their role in medical education we compared the performance and reasoning capabilities of ChatGPT-4 vs Google Gemini on nationally available subscription-based questions banks designed for helping general surgery residents prepare for the annual in-service training examination. METHODS Individual practice questions were input into GPT-4 and Gemini. Accuracy and strength of reasoning were graded on a scale of 0-3 by an individual grader for each question. A comparison of mean scores for accuracy and strength of reasoning was performed. RESULTS A total of 2516 and 2500 questions were evaluated by GPT-4 and Gemini Bard, respectively. GPT-4 answered a significantly greater number of questions correctly at 77.56% of questions correctly (86.80% easy, 75.65% medium, and 64.79% hard questions) compared to Gemini, 52.95% of questions answered correctly (58.38% easy, 52.32% medium, and 35.85% hard), p