Multimodal Emotion Classification from Audio and Video Using Hybrid Deep Learning Architectures on the CREMA-D Dataset

Emotion recognition is important for improving human computer interaction. In this study, we proposed two hybrid deep learning architectures are to recognize and classify six emotions such as anger, disgust, fear, happy, neutral, and sad using both audio and video data from the CREMA-D dataset.The first model uses a late fusion approach that combines ResNet-18 for extracting features from audio spectrograms, Vision Transformer (ViT-Tiny) for extracting features from video frames, and LSTM to learn the temporal patterns in video sequences. The second model replaces ResNet-18 and ViT with EfficientNet-B0 for both audio and video feature extraction, followed by LSTM for temporal learning and feature fusion.The results show that the EfficientNet-based model performs better, achieving an accuracy of 82%, while the ResNet18–ViT–LSTM model achieved 79% accuracy. The models performed very well in recognizing the happy emotion, but emotions like fear and sad were more difficult to classify.Overall, the results demonstrate that combining audio and visual information with temporal modeling can significantly improve emotion recognition performance.