A Survey on Real-time Multimodal Emotion Detection using Deep Learning

While human-to-human communication relies naturally on the interpretation of facial expressions and vocal nuances, detecting emotional states in Human-Computer Interaction (HCI) presents significant technical challenges. This survey paper explores the development of an integrated Emotion Recognition System (ERS) that leverages both Speech Emotion Recognition (SER) and Facial Emotion Recognition (FER) to bridge this gap. We analyze the effectiveness of deep learning architectures, specifically a hybrid CNN+BiLSTM model, in processing multimodal inputs for real-time applications. The study reviews the role of data augmentation techniques—such as noise addition and spectrogram shifting—in improving model robustness across benchmark datasets including TESS, EmoDB, and RAVDESS. By synthesizing facial and vocal features, the proposed framework aims to enhance the naturalness of human-machine communication and provide a foundation for intelligent systems in mental health, customer service, and robotics.