BCMT: Bidirectional Cross-Modality Transformer for Remote Sensing Image Captioning

Remote sensing image captioning (RSIC) focuses on generating natural language descriptions of the objects seen in remote sensing images. However, inefficient use of object texture and semantic features in images with insufficient cross-modal alignment among images and text features affects a model’s ability to generate high quality and meaningful captions. In order to address this challenge, this research proposes a Bidirectional Cross-Modality Transformer (BCMT) for RSIC to ensure the generation of meaningful captions. The BCMT tackles inefficient usage of object textures and semantic features by efficiently applying bidirectional interaction and cross-attention mechanisms to image and text features. It dynamically focusses on relevant parts of both modalities and ensures fine-grained relationships with meaningful feature utilization. Furthermore, the BCMT incorporates multi-level feature fusion which integrates high-level semantics and low-level textures for producing high-quality and context-aware captions. The University of California Merced (UCM), Sydney, and Northwestern Polytechnical University (NWPU) datasets are considered in this research. The images from these datasets are directly fed for image captioning as the dataset contains high resolution images that do require preprocessing. The BCMT introduces bidirectional cross-modality attention, cyclic semantic alignment by the Joint Representation Translator (JRT), multi-level feature fusion to enhance visual-textual interactions which provides BCMT to provide more context-aware and semantical consistent captions among UCM, Sydney and NWPU datasets. The BCMT achieves a B-1 of 95.19%, 92.18% and 94.28% respectively on the UCM, Sydney and NWPU datasets which is better than existing RSIC models.