SAM-SS: Straightforward and Efficient Designs Based on Segment Anything Model for Semantic Segmentation

Image segmentation is a fundamental task in computer vision and computational social systems, with semantic segmentation aiming to assign each pixel to a corresponding label. Due to the inherent richness of categories and contextual information in images, image segmentation remains a challenging problem. Currently, semantic segmentation models based on the segment anything model have demonstrated promising results. However, they continue to encounter challenges related to training strategies and prompt information generation. To address these issues, we propose a straightforward and efficient design method for semantic segmentation based on a prompt-free model, named SAM-SS. First, we introduce the class prompt encoder, which generates category prompts for the mask decoder to extract category-specific semantic information. Second, we incorporate the deep fusion module to bridge the semantic gap for achieving robust representation. Additionally, we observe that fine-tuning the image encoder via low-rank adaptation often leads to suboptimal convergence. To mitigate this, we propose a learning rate modulation strategy to stabilize training and boost model performance. Finally, we validate our model’s performance on three publicly available datasets. Specifically, on the Cityscapes validation set for natural images, our model achieves a mean intersection over union (mIoU) of 85.28%. Moreover, our model demonstrates strong adaptability to remote sensing imagery, achieving mIoU scores of 54.46% on LoveDA and 80.8% on the ISPRS Potsdam validation sets. These results underscore its potential utility across diverse applications.