Factors Shaping Trust and Satisfaction With AI Medical Chatbots: A Mixed Methods Vignette Survey of Caregivers Seeking Guidance on Pediatric Infectious Diseases

Abstract Background As artificial intelligence (AI) chatbots become an increasingly common source of quick medical guidance, it is important to understand whether their responses meet users’ needs and support well-informed health decisions. Yet, existing evaluation frameworks rely primarily on expert-defined evaluation dimensions that have not been empirically validated with end users. It remains unclear whether these frameworks capture the criteria people actually use when judging a response to be useful, trustworthy, and satisfying, or which evaluation dimensions matter most to users in practice. Objective This study empirically examined how commonly used dimensions such as Accuracy and Comprehensiveness shape caregivers’ satisfaction with AI-generated answers to pediatric health questions. We further investigated what expectations and communication needs may be overlooked by current evaluation frameworks. Methods We conducted a mixed methods vignette survey with 191 caregivers recruited through Prolific. Participants evaluated GPT-4o responses to a set of clinician-approved pediatric health questions across 8 dimensions and rated overall satisfaction. We quantified the influence of each dimension on overall satisfaction using a Cumulative Link Mixed Model and performed an inductive thematic analysis of open-ended comments to identify gaps in established frameworks. Results A total of 191 caregivers evaluated 1146 chatbot responses. Initially, caregivers rated Accuracy and Credibility as the most important dimensions. However, Cumulative Link Mixed Model analysis identified Usefulness as the strongest driver of overall satisfaction (odds ratio [OR] 2.53, 95% CI 1.95‐3.27; P