Learning Modality Geometry for Multispectral Object Detection: Adaptive Spacing and 3-D Rotary Position Embedding

Positional encoding is crucial for the Transformer to effectively process multi-modal feature information in multispectral object detection. However, existing studies often directly apply single-modal encoding designs to multi-modal tasks, resulting in positional encoding focusing on spatial information while neglecting modality-specific information, which limits the performance of the Transformer. To explore the potential of positional encoding in jointly processing spatial and modality information in the multi-modal domain, we propose AMRoFormer, an adaptive multi-modal rotary position embedding Transformer for multispectral remote sensing object detection. It efficiently exploits multi-modal spatial and modality information. The AMRo-Former consists of the multi-modal rotary position embedding cross attention (MRoCA) and adaptive inter-modal spacing module (AISM), where MRoCA includes multi-modal rotary position embedding (MRoPE) and cross-attention mechanism. We introduce a 3D positional encoding strategy that addresses the challenges associated with 2D positional encoding, particularly the difficulty of retaining modality-specific information. We design MRoPE by applying the 3D positional encoding strategy and rotary position embedding (RoPE), which enables the independent processing of modality information through separate 2D encoding and mixed 3D encoding. Based on MRoPE, we further introduce MRoCA, the first multi-modal attention mechanism to integrate a 3D positional encoding strategy and to introduce the concept of inter-modal spacing. To better exploit inter-modal spacing while reducing the costs associated with manually setting, we design a lightweight AISM. Integrated into a dual-stream backbone network, our method achieved state-of-the-art (SOTA) performance in multispectral object detection, attaining the highest detection evaluation metrics to date on the VEDAI, Drone-Vehicle and LLVIP datasets, with mAP50 scores of 0.8962, 0.8531 and 0.9937. Overall, AMRo-Former demonstrates exceptional generalization performance in multispectral object detection.