Comparison Of The Performance Of Chatgpt-3.5, Chatgpt-4o And Gemini 2.5 İn The Management Of Rheumatoid Arthritis: Analysis Of EULAR 2022 Recommendations

Background: Clinical guidelines are essential for diagnosing and managing rheumatic conditions like rheumatoid arthritis, which can result in deformities and impairments. Large language models such as ChatGPT and Gemini have become widely popular due to their fast responses, user-friendly interfaces, and ability to improve through human feedback and reinforcement learning. Prior studies have examined how accurate and reliable these models are when answering common public health questions. Objective: The aim of this research is to assess and contrast the answers provided by different large language models to open-ended questions based on the 2022 EULAR guideline. Materials and Methods: This was a comparative, cross-sectional study. Five overarching principles and eleven recommendations from the 2022 EULAR guidelines were rephrased into open-ended clinical questions. These questions were presented to ChatGPT-3.5, ChatGPT-4o, and Gemini 2.5 Flash. The responses were assessed using a 7-point Likert scale for usability and reliability, readability scores (Flesch-Kincaid Reading Ease (FKRE) and Flesch-Kincaid Grade Level (FKGL)), and similarity metrics (Universal Sentence Encoder (USE) and ROUGE-L). Different models were compared statistically and p < 0.05 was revealed as statistically significant. Results: Gemini 2.5 produced responses with a significantly higher FKGL than both ChatGPT-3.5 and ChatGPT-4o (p = 0.003 and p = 0.004). FKRE was significantly higher for ChatGPT-3.5 than Gemini 2.5 (p = 0.011), whereas the difference between ChatGPT-4o and Gemini 2.5 was not significant. Reliability did not differ between models (p > 0.05), but ChatGPT-4o had a significantly higher usefulness score than Gemini 2.5 (p < 0.05). ChatGPT-4o and Gemini 2.5 showed significantly higher semantic similarity than ChatGPT-3.5 (p < 0.05). ROUGE-L F1 scores showed no significant differences (p > 0.05). FKRE and FKGL were strongly negatively correlated, while reliability and usefulness were strongly positively correlated (p < 0.05). Conclusions: It was determined that ChatGPT-3.5 produced more readable responses than Gemini 2.5, while ChatGPT-4o demonstrated greater usefulness in answers to open-ended questions derived from the EULAR 2022 guideline; however, no difference in reliability was observed.