Reliability-Aware Multimodal Emotion Recognition for Heterogeneous and Corrupted Audio–Visual Environments

Multimodal emotion recognition systems often assume clean and synchronized audio–visual inputs, which limits their applicability in real-world scenarios where modalities may be degraded or unreliable. This paper introduces RAMER, a reliability-conscious multi-modal emotion recognition system that is capable of functioning with heterogeneous and corrupted environments. The proposed method is a dynamic estimate of modality reliability, and it dynamically combines audio and visual representations to ensure that the performance is stable when either or both modalities are impaired. Experiments are conducted using CREMA-D for training and RAVDESS for zero-shot cross-dataset evaluation across five shared emotion classes, with all main performance values reported as averages over three random seeds. The results show that RAMER achieves the competitive augmented cross-dataset performance, obtaining 0.461 accuracy, 0.403 Macro-F1, 0.605 macro-precision, and 0.461 macro-recall. Under clean training, RAMER achieves the highest accuracy, precision, and recall, while Early Fusion obtains the highest Macro-F1. Extreme stress testing further shows that RAMER is particularly robust under audio-only degradation, achieving the highest Macro-F1 across all evaluated audio corruption conditions. However, performance under video-only and combined audio-video corruption is more mixed, indicating that severe simultaneous degradation remains challenging. The findings demonstrate that reliability-aware fusion improves robustness in several heterogeneous conditions, while also highlighting the need for stronger adaptive mechanisms under multi-modal corruption.