Can Conversational AI Reliably Assess Depressive Symptoms? A Feasibility, Acceptability and Bias-Aware Validation Study of MADRS-Chat

Abstract Structured clinical interviews such as the Montgomery-Asberg Depression Rating Scale (MADRS) are central to depression assessment, yet their routine use is limited by time constraints and variability in administration. Conversational large language models may support standardized symptom assessment, but their feasibility, validity, and interaction-related biases in clinical settings remain unclear. We evaluated MADRS-Chat , a text-based chatbot designed to administer the MADRS interview and generate symptom-level scores using a clinician-trained scoring model. In a prospective observational study of psychiatric in- and outpatients, we assessed patient and clinician acceptability, feasibility of chatbot-administered interviews, and agreement between chatbot-derived MADRS scores, clinician-rated MADRS scores, and patient self-rated MADRS-S scores. MADRS-Chat was well accepted by both patients and clinicians, and chatbot-derived scores showed meaningful agreement with human ratings. Qualitative feedback suggested that the anonymous, written format facilitated more open and detailed disclosure of depressive symptoms. However, chatbot-administered interviews systematically yielded higher severity scores than clinician ratings, indicating that conversational format and perceived anonymity may shift patient response style. This bias was substantially reduced through lightweight post-hoc calibration. These findings suggest that conversational AI can support structured psychiatric assessment in real-world clinical care when deployed within clearly defined clinical boundaries. They also show that the mode of symptom elicitation can itself influence reported severity, underscoring the need for explicit validation, calibration, and bias assessment before clinical implementation.