Modality Interaction Decoupling: Spatiotemporal Memory-Driven Unified Multimodal Tracking Framework

Current designs of multimodal tracking networks primarily focus on spatial feature interaction within the backbone and lack the exploitation of temporal information. Although some approaches incorporate temporal cues by introducing sequential information from adjacent frames or employ updated temporal features during the feature extraction stage, they struggle to capture dynamic object variations and motion information in complex scenarios. To address these limitations, this paper proposes a Spatiotemporal Memory-Driven Unified Multimodal Tracking Framework (SMMTrack). Unlike existing works relying on feature interaction paradigms within the backbone network, this study innovatively introduces a decoupled backbone feature extraction framework. It deploys parallel, independent Vision Transformer (ViT) networks dedicated to extracting information from RGB and X modalities (RGB-T, RGB-D, RGB-E), abandoning the conventional intra-backbone feature interaction. Furthermore, we introduce a memory mechanism during the feature extraction stage to enable long-term object modeling. In addition, a Long-term Memory storage and retrieval module is designed to dynamically update the Memory-list, thereby allowing the model to capture object appearance variations and motion trends comprehensively. SMMTrack is a unified framework across three tasks (RGB-T, RGB-D, and RGB-E tracking). Experimental results demonstrate that SMMTrack outperforms the state-of-the-art (SOTA) models, achieving outstanding performance in diverse multimodal tracking scenarios. Codes and results are released on https://github.com/qfxb/SMMTrack.