Retrospective Quality Analysis of a Clinical RAG Chatbot: Observable Signals and Lessons Learned

Abstract Retrieval-augmented generation (RAG) is increasingly adopted to ground clinical conversational agents in external knowledge sources, yet many deployed prototypes lack the observability required for standard RAG evaluation. In particular, retrieved documents and grounding context are often not logged, preventing direct assessment of retrieval quality and faithfulness. We report a post-hoc evaluation of EMSy , a clinical RAG-based chatbot prototype, based on 2,660 multi-turn conversations collected between January and September 2025. Rather than benchmarking performance, we adopt an evaluation strategy based exclusively on observable signals. The analysis combines an exploratory intent analysis conducted on a random subset of heterogeneous interactions, automated quality scores available at the message and conversation level, and explicit user feedback, with 96.0% of rated conversations receiving positive feedback. Results indicate that message-level minimum scores capture localized low-quality responses that are not reflected by average conversation-level metrics, while user feedback reflects aggregate interaction impressions. This case study illustrates how diagnostic insights can be obtained under limited observability and identifies implications for the design and evaluation of future clinical RAG systems.