Facial emotion recognition and summarization in mock interviews using CNN-transformer hybrids

Facial Emotion Recognition (FER) is essential for assessing nonverbal behaviour in mock interviews, providing insights into candidates confidence, engagement, and communication effectiveness. This study uses both the FER2013 dataset and a real-world mock interview dataset, where facial regions extracted through an MTCNN-based preprocessing pipeline to ensure reliable input quality. Several deep learning models such as EfficientNet-B0, ResNet-50, CNN+LSTM, and attention-based variants were evaluated, revealing strong performance for CNN and LSTM hybrids but limitations in CBAM-enhanced models. The proposed hybrid CNN–Swin Transformer achieved the highest accuracy of 89.6%, demonstrating its advantage in combining local spatial features with global contextual modelling. Guided rubrics were defined using primary emotions and those metrics were computed to assess candidate composure and engagement. These metrics scores were further used by a LLaMA-based language model to generate personalized, rubric-driven feedback along with the improvements. The work establishes a reproducible benchmark and an end-to-end framework for automated mock interview evaluation.