Unveiling Spectral Attention Redundancy With Explainable AI

Hyperspectral image (HSI) classification has advanced significantly through transformer-based models that leverage self-attention (SA) to capture global spectral dependencies. However, most existing HSI transformers implicitly assume that SA provides a faithful and discriminative modeling of spectral information, an assumption that has rarely been examined. We examine SA behavior in HSI classification using adversarial perturbations guided by explainable AI (XAI) techniques. By comparing transformer attention distributions with Integrated Gradients (IG) importance scores, we uncover discrepancies between attended bands and decision-driving features. Experiments show that SA often forms redundant clusters across neighboring spectral bands and that attention patterns evolve across layers, shifting from a localized to a broader focus. Such redundancy improves robustness to spectral corruptions but reduces feature specificity and leads to repeated processing of highly similar bands, increasing computational burden. Our findings question the everyday use of raw attention as an explanatory clue and guide the design of more reliable and interpretable attention mechanisms for HSI.