SAM-PTx: Text-Guided Fine-Tuning of SAM With Parameter-Efficient, Parallel-Text Adapters

The Segment Anything Model (SAM) has demonstrated impressive generalization in prompt-based segmentation. Yet, the potential of semantic text prompts remains underexplored compared to traditional spatial prompts, such as points and boxes. This paper introduces SAM-PTx, a parameter-efficient approach for adapting SAM using frozen CLIP-derived text embeddings as class-level semantic guidance. Specifically, we propose a lightweight adapter, the Parallel-Text Adapter, that injects text embeddings into SAM’s image encoder, enabling semantics-guided segmentation while keeping most of the original architecture frozen. Through supervised experiments and ablations on the low-data subsets of COCO and ADE20K as well as the COD10K dataset, we show that incorporating fixed text embeddings as additional inputs improves segmentation performance over purely spatial prompt baselines. To our knowledge, this is the first work to use text prompts for segmentation on the COD10K dataset. These results suggest that integrating semantic conditioning into SAM’s architecture offers a practical, scalable path to efficient adaptation with minimal computational complexity.