Dual-Stream Masked Autoencoder-Based Method for Hyperspectral and SAR Image Classification

Land-cover classification from multi-source remote sensing imagery requires the effective integration of complementary spatial and spectral information across heterogeneous modalities, which is crucial for accurately capturing complex landscape features that cannot be fully represented by a single modality. However, conventional masked image modeling frameworks such as the Masked Autoencoder (MAE) exhibit limited capability in capturing cross-modal dependencies and often fail to preserve fine-grained structural details. To overcome these limitations, this study proposes a Dual-Stream Masked Autoencoder (DS-MAE), an attention-driven pretraining–training framework designed for joint spatial–spectral representation learning and multi-source image classification. The model incorporates a dual-stream deformable-attention Transformer to separately reconstruct masked spatial pixels and spectral channels, while Swin-Transformer branches along spatial and spectral directions enhance sensitivity to small objects, high-frequency textures, and object boundaries. A cascaded feature fusion module (CFM) further strengthens cross-modal alignment by effectively integrating global and local information from different modalities. Experiments conducted on Berlin, Augsburg, and Houston 2018 datasets demonstrate that DS-MAE consistently surpasses seven state-of-the-art methods in Overall Accuracy, Average Accuracy, and Kappa coefficient, with particularly notable gains for classes exhibiting complex spatial structures such as buildings and roads. These results confirm the robustness and generalizability of DS-MAE and highlight its effectiveness in capturing complementary spatial–spectral cues for advanced multi-source remote sensing image classification.