It’s Not Just for Trust: Designing for Emerging Uses of Explainable AI in Clinical Decision-Making

Explainable AI (XAI) is often viewed as a mechanism to promote the transparency and interpretability of AI recommendations in high-stakes domains, such as healthcare. This has led many studies to focus on designing and evaluating XAI to foster trust, calibrate reliance, enable algorithmic recourse, and support model understanding. However, this limited design scope restricts our understanding of how XAI can be used more broadly to support critical tasks in complex workflows, such as facilitating shared decision-making and supporting communication between stakeholders. Our work aims to address these critical gaps in the design for and understanding of XAI's emerging uses by iteratively prototyping XAI designs for an AI-powered clinical decision-support system with clinical stakeholders. We then created a high-fidelity prototype from those iterative sessions and used it as a design probe to uncover four emerging uses of XAI: collaboratively exploring treatment options, identifying and reflecting on treatment plans, communicating with stakeholders, and supporting health education. We reflect on the implications of designing XAI for emerging uses in healthcare.