GSMFormer: A Structure-Aware Framework with Agent-Gated Fusion for Building Semantic Segmentation

Building semantic segmentation from very-highresolution (VHR) remote-sensing imagery remains challenging due to background clutter and large-scale variations. Existing approaches often lose boundary details during decoding, whereas many boundary-aware methods rely on non-learnable operators, which limits their adaptability and introduces irrelevant background noise. Moreover, the lack of adaptive semantic modulation in multi-scale fusion can cause semantic misalignment. To address these issues, this paper proposes GSMFormer, an end-to-end RGB-only framework that integrates learnable geometric priors with semantic consistency. Specifically, we introduce a lightweight Structure-Aware Dynamic Geometric Prior (S-DGP) module, explicitly encoding structural priors to enhance the discriminability of building features against complex backgrounds. To resolve the feature misalignment in multi-scale fusion, this paper designs an Agent-Gated Semantic Fusion (AGSF) mechanism, which leverages global agents as semantic bridges to dynamically align heterogeneous features, ensuring that multi-scale contexts are harmoniously integrated under high-level semantic guidance. To rebuild the extracted feature maps, this paper proposes the Hierarchical-Attention Head (HAH), leveraging differential signals and multi-scale features to enhance the perception of building roofs. Experiments on the WHU building dataset, the Massachusetts building dataset, and the ISPRS Potsdam dataset demonstrate the state-of-the-art performance, achieving 90.72%, 70.56%, and 92.22% on IoU, respectively. Codes are available at https://github.com/Hitori-Bocchi/GSMFormer.git . Index Terms —Building Extraction, Semantic Segmentation, Transformer, Structure-Aware, Multi-scale Fusion.