Learning Semantic-level Multi-modal Alignment Transformer for RGB-D Co-salient Object Detection

The goal of RGB-D Co-Salient Object Detection (CoSOD) is to integrate both depth and RGB information to identify common and salient objects across a set of images. However, the existing leading research paradigm often fuses raw depth and RGB cues indiscriminately via simple concatenation or weighted summation, neglecting a fundamental limitation: the low inter object contrast in depth maps often causes misalignment between RGB-salient regions and depth-highlighted areas, resulting in model performance degradation. To address this discrepancy, we propose a Semantic-level Multi-modal Alignment transFormer (SMAFormer) that learns to establish precise correspondence for the co-salient objects across RGB and depth modalities. The SMAFormer mainly incorporates two key components: a Learnable Depth-calibration Module (LDM) and a cross-modal transformer (CMT). In the LDM, a Semantic-level Cluster center Extractor (SCE) is first designed to generate semantic level masks, which are then converted to semantic-level cluster centers via the Segment Anything Model (SAM). Afterwards, a Recurrent Deep Clustering Module (RDCM) iteratively projects these semantic representations onto the depth features through a learnable clustering strategy, accurately calibrating the depth features with rich semantic alignment information between two modalities. Next, in CMT, cross-modal shared representations between RGB and calibrated depth features are extracted and mixed through interaction-attention mechanism. Extensive evaluations demonstrate that SMAFormer establishes new state-of the-art performance across three benchmarks including RGB-D CoSal150, RGB-D CoSeg183, and RGB-D CoSal1k.