Does Class Imbalance Undermine SHAP Explanations? A Multi-Domain Empirical Study of Feature Attribution Stability

Abstract SHAP (SHapley Additive exPlanations) is the most widely used post-hoc explanation method for machine learning models, yet its reliability under class imbalance—the dominant condition in real-world classification tasks—remains poorly understood. This study presents a systematic empirical evaluation of how class imbalance affects SHAP explanation stability across 10 benchmark datasets (8 domains), 6 model architectures, 7 imbalance ratios (1:1 to 1:100), and 6 mitigation strategies. This study proposes a five-dimensional evaluation framework assessing rank stability, Top-K overlap, magnitude consistency, direction fidelity, and value divergence. Results reveal three key findings. First, models trained on imbalanced data produce SHAP explanations that diverge significantly from balanced-data references across all five dimensions, with effects detectable at ratios as mild as 1:2 (Wilcoxon p < 0.001, Cohen’s d = 0.813 to 2.134). Second, standard resampling techniques (SMOTE, ADASYN, random undersampling) counterintuitively increase this divergence in 56–68% of cases, while class weighting—the only strategy that does not actively worsen stability—produces a practically negligible improvement (+0.018 in rank stability). Third, explanation dimensions differ markedly in sensitivity: Top-K feature overlap drops to 0.52 at 1:100, while magnitude consistency remains above 0.86. These findings show that class imbalance alters the feature relationships models learn, and that SHAP faithfully captures these altered relationships—producing explanations that may mislead practitioners who assume invariance across class distributions. Practical guidelines are provided for obtaining more trustworthy explanations from imbalanced datasets.