BoostTransformer: A Lightweight Transformer for Remote Sensing Image Captioning

Remote sensing image captioning (RSIC) aims to translate remote sensing images (RSIs) into textual descriptions, where Transformer-based approaches have attained remarkable performance. However, the prohibitive computational costs of self-attention and cross-attention on high-resolution features compel existing RSIC methods to rely on pre-trained backbones or prerequisite tasks to compress input RSIs, resulting in the loss of multi-scale semantic information and hindering end-to-end training and inference. To address this challenge, we propose an Attention Matrix Expansion Mechanism, which endows the approximation of large-scale global self-attention and cross-attention with negligible computational overhead (e.g., only +0.173 GFLOPs). Building on this mechanism, we design BoostTransformer, a RSIC framework constituted by two core components: BoostEncoder, a lightweight Transformer encoder that directly processes raw RSIs to achieve panoramic feature extraction, and BoostDecoder, a lightweight Transformer decoder that extensively integrates multi-scale encoder features, thereby boost the quality of generated captions. We further derive a training paradigm from experiments for end-to-end training RSIC model from scratch. Extensive experimental results demonstrate our end-to-end lightweight model achieves competitive performance without pre-training. The code will be available at https://github.com/ACT-WAVE/BoostTransformer.