Evaluating the Accuracy, Readability, and Consistency of Artificial Intelligence Models in Patient Education for Rheumatoid Arthritis, Osteoarthritis, and Psoriatic Arthritis

Read full text Summarise with VIRAT
Authors: Morgan Lilly, Kanika Monga
Cited by: 0
References: 16

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

INTRODUCTION AND AIM: Artificial intelligence (AI) chatbots are increasingly used by patients to obtain medical information before seeking clinical care; however, the accuracy, readability, and consistency of AI-generated information in rheumatology remain uncertain. This study aimed to assess the readability, accuracy, and consistency of large language models (LLMs) generated patient education content for rheumatoid arthritis (RA), osteoarthritis (OA), and psoriatic arthritis (PsA). METHODS: From August 18, 2025, to August 24, 2025, three standardized patient-facing questions per disease were submitted daily to three LLMs, specifically ChatGPT (San Francisco, CA: OpenAI), Google Gemini (Mountain View, CA: Google LLC), and OpenEvidence (Cambridge, MA: OpenEvidence Inc.), with histories cleared between submissions. Readability (Hemingway grade level), word count, accuracy (on a 1-5 scale), and day-to-day consistency (Jaccard similarity) were measured. Responses from each model's most consistent day were accuracy-rated by rheumatology fellows and attendings. RESULTS: A total of 189 responses were collected. No model consistently met the American Medical Association (AMA)/National Institutes of Health (NIH)-recommended reading levels for sixth through eighth grade. OpenEvidence produced the most technical content (≥17th grade, indicating postbaccalaureate readability), while ChatGPT and Gemini averaged 11.9-12.0 grade. Gemini generated the longest responses (>500 words). ChatGPT showed the highest day-to-day stability (range:

Fields of Study & Subjects

Artificial Intelligence in Healthcare and Education Health Literacy and Information Accessibility Social Media in Health Education

Publication details

Journal / source
Cureus
Volume
18
Issue
3
Pages
e106160–e106160
Publication date
30 March 2026
Language
EN