Semantic-Aware Remote Sensing Visual Question Answering via Segmentation-Guided Learning

Visual question answering (VQA) in remote sensing has emerged as a challenging yet promising research area due to the complexity of high-resolution satellite imagery, diverse land cover distributions, and the need for effective multi-modal fusion. In this paper, we propose a novel remote sensing VQA framework that leverages semantic segmentation results to enhance the extraction and focus of critical visual features. Our model employs a dual-branch feature extraction strategy: one branch captures pixel-level features using a convolution neural network (CNN) backbone, while the other utilizes a graph neural network (GNN) to model the relationships among segmented regions, thereby extracting high-level, category-specific semantic information. In addition, an auxiliary contrastive learning module is integrated to align key terms in the question with corresponding regions in the segmentation map, effectively distinguishing foreground from background elements and reinforcing the model’s attention on question-relevant areas. Extensive experiments on the EarthVQA, RescueNet-VQA, and FloodNet-VQA datasets demonstrate that our approach significantly outperforms state-of-the-art methods in terms of VQA accuracy. Our results highlight the benefits of incorporating semantic segmentation as a prior, offering a comprehensive and interpretable solution that lays the groundwork for future research in remote sensing VQA.