On the Design and Evaluation of Human‐Centered Explainable AI Systems: A Systematic Review and Taxonomy

As Artificial Intelligence (AI) becomes more common in everyday living, there is an increasing demand for intelligent systems that are both performant and understandable. Explainable AI (XAI) systems aim to provide comprehensible explanations of decisions and predictions. At present, however, evaluation processes are rather technical and not sufficiently focused on the needs of human users. Consequently, evaluation studies involving humans can serve as a valuable guide for conducting user studies. This paper presents a comprehensive review of 65 user studies evaluating XAI systems across different domains and application contexts. As a guideline for XAI developers, we provide a holistic overview of the properties of XAI systems and evaluation metrics focused on users (human centered). We propose goals for the human‐centered design (design goals) of XAI systems for different user groups (AI novices and data experts). The first part of our results includes the analysis of XAI system characteristics. Importantly, we distinguish between the core system and the XAI explanation, which together form the whole system. Further results include the distinction of evaluation metrics into affection, cognition, usability, interpretability, and explanation metrics. Furthermore, the users, along with their specific characteristics and behavior, can be assessed. For AI novices, the relevant extended design goals include responsibility, acceptance, and user experience. For data experts, the focus is performance oriented, including human‐AI collaboration and system task performance. Future literature trends include the design of context‐aware and inclusive systems, adapting to users and the situational context. Several limitations in the reviewed literature were identified: a lack of validation, standardization, and documentation; evaluation is focused on particular metrics, rather than holistic; evaluation of explanatory and transparency components of the system is often neglected in user studies. These research gaps could be addressed by integrating existing XAI evaluation frameworks and developing validated questionnaires.