CLIP-Guided Dynamic Feature Fusion With Relation-Enhanced Attention for Remote Sensing Image Captioning

Remote Sensing Image Captioning (RSIC) is a multimodal task that aims to automatically generate natural language descriptions for remote sensing images. However, the inherent scale diversity and semantic complexity of remotely sensed images still pose significant challenges in the semantic alignment of image-text and explicit modeling of complex scene structures. To solve these problems, a new CLIP-guided dynamic feature fusion and attention-based relation-enhanced model is proposed, which is referred to as CDRE. Specifically, unlike directly using the CLIP model pre-trained on natural images, we first fine-tune the encoder of CLIP on remote sensing image datasets, thereby significantly improving its capability of image-text semantic alignment in remote sensing scenes. Subsequently, we employ a feature extraction (FE) module to obtain a set of region features and the fine-tuned CLIP image embeddings. In addition, we design a dynamic semantic fusion (DSF) module, which adaptively integrates the image embeddings and region features to generate hierarchically and semantically consistent image descriptions. Meanwhile, to enable explicit modeling of complex relationships in the scene, we introduce the relation-enhanced attention (REA) mechanism by combining 14 predefined spatial orientation relation terms, which effectively capture and represent fine-grained spatial and semantic dependencies among different objects in the image. Finally, a Transformer integrated with the REA mechanism and multi-head cross-attention (MCA) is employed to enable deep interaction between visual and textual information, ultimately generating the final descriptive text. Extensive experimental results on three public remote sensing image captioning datasets (RSICD, UCM-Captions, and Sydney-Captions) demonstrate the superior performance of CDRE.