RAG in clinical practice: a cautionary tale of AI ‘Truthfulness’
Abstract
Retrieval-augmented generation (RAG) aims to curb large language models (LLMs) hallucinations, yet its conversational reliability is uncertain. We tested a clinical RAG by executing the same query 100 times at varying dialogue lengths. The hallucination rate surged from 5%(no history) to 40% with just 10 prior exchanges, revealing a critical failure mode. Rigorous conversational testing is essential for patient safety before clinical deployment of RAG systems.
Fields of Study & Subjects
Artificial Intelligence in Healthcare and Education
Machine Learning in Healthcare
Topic Modeling
Publication details
- Journal / source
- npj Health Systems
- Publisher
- Springer Science and Business Media LLC
- Volume
- 3
- Issue
- 1
- Publication date
- 3 July 2026
- Language
- EN