RTMT-Net: A Real-Time Transformer-Based Multimodal Framework for Emotion Recognition Using Facial, Speech, and EEG Signals

Emotion recognition systems face challenges in real-world variability, where single modalities like facial cues or voice falter under noise or occlusion. This study introduces RTMTNet, a real-time transformer-based framework that integrates facial expressions, speech prosody, and EEG signals for robust detection of discrete emotions (happy, sad, fear, calm) and dimensional valence-arousal, validated against self-reported labels from benchmark datasets such as DEAP and MAHNOB-HCI [1]. Facial features employ Vision Transformer (ViT) for spatial embeddings, speech prosody uses Conformer on log-Mel spectrograms capturing pitch and rhythm, and EEG leverages temporalspatial transformers on differential entropy features from 32 channels [2][3]. A cross-modal fusion module with multi-head self-attention dynamically weighs modalities, modeling long-range dependencies for superior fusion. Trained on 1,200+ trials from 40 subjects (80/10/10 train/validation/test split, subject-independent), RTMT-Net achieves 86.4% accuracy, F1=0.85, and concordance correlation coefficient (CCC) = 0.81 for valence-arousal, surpassing unimodal baselines (face: 78%, speech: 73%, EEG: 75%) and late-fusion CNNs (81%) by 5-11% [1][4]. Real-time inference reaches 52 FPS on NVIDIA Jetson edge hardware, with ablation studies confirming EEG's value in ambiguous audiovisual contexts [5].