Multimodal emotion recognition based on combined deep learning networks

Aiming to address the issues of low accuracy, low F1 score, and long task completion time in traditional multimodal emotion recognition methods, a multimodal emotion recognition method based on a combined deep learning network is proposed. Firstly, EEG signals, eye movement data, and facial expression images are collected, and the features of the collected data are extracted. Then, using the modal attention module, weighted operation, and decision module, a multimodal feature fusion model is built, and the extracted features are used as model inputs to obtain the multimodal feature fusion results. Finally, the results of multimodal feature fusion with a combined deep learning network are combined to achieve multimodal emotion recognition. The experimental results show that the proposed method has a maximum accuracy of 99.1% for multimodal emotion recognition, a minimum F1 value of 0.947, and a minimum completion time of 56.8 ms for multimodal emotion recognition tasks, demonstrating high precision and efficiency.