Reference and Attention Guided Few-Shot Adaptation of Segment Anything Model for Remote Sensing Images

The Segment Anything Model (SAM) has demonstrated remarkable performance as a zero-shot segmentation framework. However, in the field of remote sensing, it faces two major challenges: (1) SAM often underperforms in remote sensing images (RSIs) with significant domain gaps. Existing mainstream approaches typically rely on supervised model adaptation, which requires a large amount of annotated data and results in high costs; (2) Although SAM supports flexible geometric prompt inputs, the ambiguity arising from a single foreground point potentially corresponding to multiple objects poses a critical challenge for segmentation in RSIs. To address these issues, we propose RefAtt-SAM. Specifically, a reference image and its corresponding mask are introduced to extract foreground features, which are then fused with the input image features to generate a foreground attention map. This map guides the network to focus on foreground regions, thereby mitigating the ambiguity introduced by single-point prompts. In addition, the proposed method incorporates high-frequency adapters into the image encoder and learnable tokens into the mask decoder. This dual integration addresses the domain gap between natural and RSIs. Experimental results demonstrate that the proposed approach significantly improves segmentation accuracy for RSIs under single-point prompt conditions. The code is available at https://github.com/WILSON10111/RefAtt-SAM.