QANet: Query-Aware Multi-Modal Prior Refinement for Few-Shot Segmentation

Recent Few-Shot Segmentation (FSS) approaches incorporate vision-language models to improve segmentation by constructing multi-modal priors, including visual, CAM and textual priors. However, these priors are often suboptimal, suffering from background interference, incomplete activation and semantic misalignment. To address these limitations, we propose a query-aware multi-modal prior refinement framework (QANet) for FSS, in which the multi-modal priors are jointly refined for more accurate segmentation by fully exploiting target-relevant cues from query features. Specifically, QANet comprises two modules, i.e., a Query-aware Multi-modal Prior Refinement (QMPR) module for visual and CAM prior refinement, and a Query-Aware Textual Embedding and Prior Refinement (QTEPR) module for textual prior refinement. More specifically, in QMPR, a query self-attention bilateral-guided enhancement strategy and a query self-similarity prior refinement strategy are carefully designed for suppressing background interferences and recovering foreground completeness, respectively. And in QTEPR, such refined visual and CAM priors are first leveraged to extract target-relevant cues for visual-textual domain gap mitigation and text embedding enhancement, producing query-aware textual priors. These textual priors are then enhanced under the guidance of those refined multi-modal priors. Extensive experiments on PASCAL-5iand COCO-20idatasets demonstrate that QANet achieves new state-of-the-art performance. Moreover, QANet remains competitive on cross-domain FSS and weak-label FSS tasks.