A Review of Taxonomies of Explainable AI (XAI) Evaluation

Research on explainable AI (XAI) has expanded rapidly, yet evidence about whether explanations are faithful, robust, or fit for purpose remains difficult to synthesize because evaluation practice is fragmented. A central obstacle is the lack of shared evaluation standards: numerous survey and review papers propose taxonomies intended to structure what should be measured, for whom, and for which purpose, but they differ substantially in scope, terminology, and underlying assumptions. In this paper, we present a systematic literature review of 170 articles that propose or employ taxonomies of XAI evaluation. We extract recurring problems that undermine comparability, including overlapping evaluation categories and inconsistent use of terms. Specifically, we find that taxonomies are dominated by a distinction between human-based and human-independent evaluation, though this distinction appears under many different names. Furthermore, we identify nearly 240 labels used for evaluated properties (around 80 recurring), revealing extensive synonymy and homonymy. We argue that the field does not need yet another taxonomy; instead, we outline how existing taxonomies can be combined and propose “property cards” as building blocks for a shared language that aligns disparate schemes and enables more comparable reporting across studies.