AMPUNet: Hierarchical Attention Map Pyramid for Semantic Segmentation of Remote Sensing Images

Accurate semantic segmentation of high-resolution remote sensing imagery is essential for applications ranging from urban planning to environmental monitoring. However, this task remains fundamentally challenging due to the complex spatial patterns, extreme scale variations, and fine-grained details inherent in geographical scenes. While attention mechanisms, particularly global and sparse attention, have shown promise in capturing long-range dependencies, existing approaches often suffer from three interconnected limitations: prohibitive computational complexity, misalignment when integrating multi-scale representations, and loss of semantic information during the decoder's upsampling stages. This paper introduces AMPUNet, a novel framework designed to overcome these limitations through the construction of a hierarchical, coarse-to-fine attention map pyramid. Our core innovation lies in explicitly propagating and refining attention maps across network layers rather than operating solely on feature maps. Specifically, we design: (1) a hybrid sparse attention framework combining a Block Attention Module and a Column Attention Module to model global context efficiently; (2) a Dimension Correspondence Module to achieve tensor-level granularity alignment for multi-scale attention maps; and (3) an Attention Map Merging Module with a Cross-layer Attention Weighting strategy, which directly transfers high-level semantic information from deep to shallow layers, mitigating information degradation. Extensive experiments on the ISPRS Vaihingen, Potsdam, and LoveDA benchmarks demonstrate that AMPUNet achieves superior performance, with mIoU scores of 75.43% on Vaihingen, 78.03% on Potsdam, and 50.94% on LoveDA, while maintaining competitive inference efficiency. Our findings confirm that structuring attention into a learnable pyramid is a highly effective paradigm for remote sensing semantic segmentation, successfully balancing precise detail preservation with robust global understanding. The source code is available at:https://github.com/interpretty/AMPUNet.