Evaluating the Accuracy, Readability, and Consistency of Artificial Intelligence Models in Patient Education for Rheumatoid Arthritis, Osteoarthritis, and Psoriatic Arthritis
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
INTRODUCTION AND AIM: Artificial intelligence (AI) chatbots are increasingly used by patients to obtain medical information before seeking clinical care; however, the accuracy, readability, and consistency of AI-generated information in rheumatology remain uncertain. This study aimed to assess the readability, accuracy, and consistency of large language models (LLMs) generated patient education content for rheumatoid arthritis (RA), osteoarthritis (OA), and psoriatic arthritis (PsA). METHODS: From August 18, 2025, to August 24, 2025, three standardized patient-facing questions per disease were submitted daily to three LLMs, specifically ChatGPT (San Francisco, CA: OpenAI), Google Gemini (Mountain View, CA: Google LLC), and OpenEvidence (Cambridge, MA: OpenEvidence Inc.), with histories cleared between submissions. Readability (Hemingway grade level), word count, accuracy (on a 1-5 scale), and day-to-day consistency (Jaccard similarity) were measured. Responses from each model's most consistent day were accuracy-rated by rheumatology fellows and attendings. RESULTS: A total of 189 responses were collected. No model consistently met the American Medical Association (AMA)/National Institutes of Health (NIH)-recommended reading levels for sixth through eighth grade. OpenEvidence produced the most technical content (≥17th grade, indicating postbaccalaureate readability), while ChatGPT and Gemini averaged 11.9-12.0 grade. Gemini generated the longest responses (>500 words). ChatGPT showed the highest day-to-day stability (range:
Fields of Study & Subjects
Artificial Intelligence in Healthcare and Education
Health Literacy and Information Accessibility
Social Media in Health Education
Publication details
- Journal / source
- Cureus
- Volume
- 18
- Issue
- 3
- Pages
- e106160–e106160
- Publication date
- 30 March 2026
- Language
- EN