Evaluating Three Large Language Models in Ophthalmology Education: A Comparative Study of Accuracy and Readability Across 442 Questions

Read full text Summarise with VIRAT
Authors: Semih Çakmak, Arzu Karakiraz, Işılsu Ezgi ULUIŞIK, Ali Efe Kara, Şerife Bayraktar, Emre Altinkurt
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

To compare the accuracy and readability of ChatGPT-4.0 Mini, Gemini 1.5 Flash and Microsoft Copilot generated responses to fifth year medical school ophthalmology exam questions. A total of 442 multiple-choice questions were submitted individually to each chatbot. Responses were marked correct or incorrect based on an answer key. Readability was assessed using Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease (FRE) and Simple Measure of Gobbledygook (SMOG) indices. There was a statistically significant difference in the overall accuracy rates among the chatbots (p < 0.001). Copilot has the highest accuracy rate (89.4%), followed by ChatGPT (84.2%) and Gemini (76.7%). Readability indices also showed a significant difference (p < 0.001 for all). Copilot demonstrated the lowest linguistic complexity, with the lowest FKGL and SMOG scores and the highest FRE values. In contrast, ChatGPT generated responses with the highest linguistic complexity across all evaluated metrics. Intraclass correlation coefficients for readability metrics ranged between 0.68 and 0.83, with the highest agreement between ChatGPT and Gemini. Despite the statistical significance in readability differences, all responses generally required a high school to early college-level reading ability. Large language model-based chatbots demonstrated variable performance in answering ophthalmology questions at the medical school level. Among the models evaluated, Microsoft Copilot performed best in both accuracy and readability. These findings suggest that model choice may influence the usefulness of AI-generated content in educational settings.

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Artificial Intelligence in Education Ophthalmology and Visual Health Research

Publication details

Journal / source
Figshare
Publication date
25 March 2026
Language
EN