Advancing Efficient Vision Foundation Models for Analysis of Multispectral Imagery

Traditional supervised learning methods are typically task-specific and highly dependent on labeled data, which limits their scalability and generalization ability in remote sensing applications. Vision foundation models (VFMs) based on self-supervised learning are highly valued in the analysis of RGB imagery for their ability to generalize well without labeled data. Foundation models are particularly suited for remote sensing applications, where datasets are large and high quality labels are limited. Despite their potential, several challenges remain: 1) Traditional masked autoencoder (MAE) techniques in multispectral data may focus primarily on reconstructing multispectral imagery information, potentially limiting their effectiveness in learning meaningful latent representations. 2) There is a lack of in-depth analysis of feature representations during self-supervised training, with evaluations often dependent on performance after fine-tuning. 3) The typically practice of full parameter fine-tuning in downstream tasks is computationally expensive. To address these challenges, a novel self-supervised VFM specifically designed for multispectral image analysis is proposed. The proposed approach utilizes latent representations from the encoder module to selectively reconstruct masked patches, reducing pretraining parameters and enhancing feature representation capabilities from the encoder. Additionally, during the pretraining phase, weighted attention distance and feature map frequency are employed to explicitly analyze feature learning across both spatial and frequency domains. Leveraging the strong generalization ability gained from pretraining on a large-scale in-domain dataset, the model requires only a lightweight fine-tuning strategy to adapt to different downstream tasks, significantly reducing computational costs. by utilizing only 5.2% of the trainable parameters in the backbone. Experiments on three downstream datasets show that the proposed model achieves competitive results in classification, with a Top-1 accuracy of 99.44% and a multi-label mAP of 88.2, as well as in segmentation, with a mIoU of 50.1, while requiring significantly fewer trainable parameters in both the pretraining and fine-tuning stages.