MERDiff-ODE

Emotion recognition (ER) plays a significant role in improving the interaction between humans and computers, as it enables systems to understand and interpret the emotions of users. This, in turn, allows the systems to adjust or react appropriately. It has a wide application in the field of healthcare. Traditional ER systems primarily relied on unimodal inputs, such as text, audio, or facial expressions, limiting their adeptness to capture the full spectrum of human emotions. However, with deep learning and advances in multimodal processing, ER systems have evolved to integrate multiple input modalities—including text, audio, video, and physiological signals—leading to more robust and accurate ER for the treatment of patients undergoing psychological and mental health issues. Despite these improvements, a significant challenge in multimodal emotion recognition (MER) is the issue of missing or incomplete data from one or more modalities, which can degrade performance. In response to this challenge, this paper presents an innovative method that leverages diffusion models for addressing incomplete modality scenarios in MER. Generative probabilistic models known as diffusion models are adept at capturing intricate data distributions, which makes them well suited for producing high-quality synthetic data to fill in missing modalities. The input random noise can be transformed by these models into the distribution space of missing modalities, enabling them to recreate the missing data while maintaining the original semantic and statistical characteristics of the inputs. The proposed method, MERDiff-ODE, not only fills in missing modalities but also uses existing modalities to guide the recovery process, reducing ambiguity between the original and recovered data. This approach not only enhances the overall performance of MER systems by ensuring a more complete and coherent multimodal input but also addresses the limitations, such as high computational demands and latency issues, which accompany the use of diffusion models. Nonetheless, MERDiff-ODE offers a promising elucidation for improving the robustness and reliability of MER in healthcare applications.