CDFIT: A Transformer Using Cross-Modal Dual-Stream Feature Interaction for Multispectral Pedestrian Detection

Modality imbalance is a significant challenge for multi-modal interaction at various depths in multispectral pedestrian detection under varying illumination environments. To overcome the limitations of current cross attention in addressing the modality imbalance, we propose the Cross-Modal Dual-Stream Feature Interaction Transformer (CDFIT). CDFIT capitalizes on the Transformer’s ability to learn long-range dependencies, extracting global intra-modal and inter-modal correlations during the feature interaction phase. Crucially, in order to effectively eliminate the interference of the self-attention within one modality to the alternative one, we propose horizontal and vertical correlation decoupling modes to divide and reassemble the attention maps in CDFIT. This facilitates more purified inter-modal attention while preserving relevant intra-modal self-attention, reducing the information interference. Meanwhile, in CDFIT, we expand Transformer into dual-stream pathways to align and assemble the information from RGB and thermal modalities across depths separately, thereby greatly enhancing the performance of multispectral object detection. Comprehensive experiments and ablation studies on benchmark datasets demonstrate that CDFIT achieves superior performance compared with state-of-the-art methods.