Meta-evaluation of robustness metrics: An in-depth analysis

Robustness is a pivotal attribute of trustworthy explainable artificial intelligence, ensuring that explanations remain consistent when inputs are perturbed slightly. Although numerous metrics have been proposed to quantify this robustness, their reliability remains largely unverified. This study introduces a novel meta-evaluation framework for robustness metrics, grounded in controlled, verifiable experimental setups. We propose three sanity tests: perfect explanation, normal explanation, and random output. These tests facilitate the systematic assessment of the validity of robustness metrics using transparent models. By evaluating seven state-of-the-art robustness metrics across four benchmark datasets, our results reveal significant shortcomings: no single metric consistently achieves the expected outcomes across all tests. These findings underscore fundamental flaws in robustness metrics and emphasise the necessity for improved evaluation frameworks. Our methodology provides a reproducible, assumption-light benchmark for identifying unreliable metrics before deployment in critical applications.