A multimodal reconstruction network via dynamic gradient modulation for emotion recognition

Emotion recognition aims to automatically analyze and identify human emotions through artificial intelligence, with broad applications in areas such as education and autonomous driving. However, this field still faces several significant challenges. On one hand, during the training process of multimodal emotion recognition models, dominant modalities often suppress the learning of other modalities. On the other hand, traditional multimodal models tend to focus on extracting joint representations for emotion classification, while overlooking the expressive capacity of each individual modality. To address these issues, we propose a novel multimodal reconstruction network based on dynamic gradient modulation, termed MRNDGM, designed for emotion recognition tasks involving video and audio modalities. By integrating modality reconstruction with emotion classification, MRNDGM not only enhances emotion prediction performance but also preserves the original characteristics and structural information of each modality. Furthermore, by dynamically adjusting gradient ratios according to inter-modal discrepancy rates, our method adaptively mitigates the optimization imbalance among modalities, thereby improving the learning of weaker modalities. Experiments conducted on real-world multimodal emotion recognition datasets demonstrate that MRNDGM outperforms existing state-of-the-art methods in terms of both accuracy and robustness, showcasing its strong potential for practical applications.