Revisiting Frequency Domain: Spatial-Frequency Joint Tuning for Referring Image Segmentation

Referring image segmentation aims at segmenting the target object referred by a natural language expression, which requires semantic-level object understanding and pixel-level contour segmentation. Existing methods are limited to the spatial domain, thus ignoring potential discriminability from the frequency domain and missing mutual boost between the spatial and frequency domains, also facing heavy high-frequency degeneration issues. In this paper, we revisit frequency domain and propose a novel lightweight spatial-frequency joint tuning (SFJT) plugin for referring image segmentation. We decompose 4 mainstream methodology paradigms into two general stages consisting of roughly semantic-level object understanding and precisely pixel-level contour segmentation, then respectively enhance them with parameter-efficient spatial-frequency tuning strategies. Specifically, we develop a spatial-frequency joint prompting technique (SFJ-Prompt) during early object understanding stage, which mines bidirectional spatial-frequency information to realize mutually spatial-frequency boosting, facilitating more comprehensive object understanding. Besides, we introduce a LoRA-based high-frequency auxiliary branch (HF-LoRA) during latter contour segmentation stage, which compensates for heavy high-frequency degeneration issues in spatial neural networks, facilitating more precise contour segmentation. Eventually, extensive experiments of 4 mainstream methodology paradigms for referring image segmentation on 4 large-scale datasets demonstrate the effectiveness and superiority of the proposed method.