Cross-Modality Spatial–Spectral Fusion Transformer With Spectral Amplification for VHR Image Classification

Very high-resolution (VHR) image classification is a key and challenging task in remote sensing. The classification performance can be effectively enhanced by mining complementary information from different modality images paired and aligned with VHR images, such as LiDAR and hyperspectral images (HSI). However, acquiring these multi-modality images requires different observation platforms and sensors, incurring substantial data collection costs. Moreover, the intrinsic difference in information between modalities is frequently overlooked, resulting in sub-optimal use of inter-modality complementary information. To alleviate the above problems, we propose a Cross-Modality Spatial-Spectral Fusion Transformer (CMS2F-Former) with Spectral Amplification to improve VHR image classification performance. Specifically, CMS2F-Former employs an adversarial spectral amplification network to learn rich spectral features from existing real HSI to generate HSI corresponding to VHR images without additional equipment. Furthermore, we design a cross-modality spatial-spectral fusion network to fully leverage spatial information and spectral features in VHR images and generated HSI. Within modality, the frequency attention spectral enhancement module is developed to highlight the spectral features that describe the detailed information of targets according to frequency intensity, while suppressing redundant information. Multiple groups of difference perception fusion mechanisms are embedded across modalities, driving fully complementary fusion of deep semantic features through the semantic discrepancies information of tokens between modalities. Our approach outperforms state-of-the-art algorithms on three sets of VHR images with limited training samples.