Multimodal Remote Heart Rate Estimation via Spatio-Temporal Transformers and Adaptive Fusion

Remote photoplethysmography (rPPG) enables non-contact heart rate estimation from facial videos. Despite recent advances, single-modality methods remain vulnerable to motion, illumination changes, and modality-specific degradations. We address these limitations with a multimodal framework that explicitly leverages complementary RGB and infrared (IR, thermal or NIR) streams. Built on a 3D SwiftFormer backbone, the method integrates three modules: (1) a Context-aware Temporal Difference Convolution (CTDC) that amplifies motion-sensitive cues via multi-scale temporal differencing; (2) a Bidirectional Cross-Attention (BCA) that enables hierarchical information exchange between modalities; and (3) a Cross-Modal Gating Fusion (CMGF) that adaptively combines features using a temperature-scaled logit-difference gate. Training is guided by a hybrid objective over time and frequency, augmented with a scheduled soft-DTW alignment term. Extensive experiments on two public datasets demonstrate consistent improvements over state-of-the-art baselines, with ablation studies confirming the contributions of CTDC, BCA, CMGF, and soft-DTW. These results highlight the effectiveness of explicit cross-modal interaction and adaptive fusion for robust, accurate remote heart rate estimation.