Explainable Multimodal AI for Communication Coaching with Emotion Recognition

View at publisher Summarise with VIRAT
Authors: Bharathi V, Prasanth Sai Chintakrindha, Venkata Prasad Akurati
Cited by: 0
References: 15

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

A multimodal explainable Artificial Intelligence (XAI) framework for self learning English communication education, particularly for interview preparation. It is primarily audio, video, and facial expression based. The solution is a combination of video based emotion recognition, lip movement analysis, and excellent fluency to understand verbal and nonverbal features. A Conventional Neural Network (CNN) removes affective elements from facial expressions, and Recurrent Neural Network (RNN) models are used for lip movement patterns to evaluate clear quality. These self supervised speech models support fluency and good scoring. These streams are combined through a Bayesian confidence estimator to derive an accurate measure of the emotion of a learner. A transformer based generator converts analytical outputs into personalized feedback. Experimental evaluation gains good quality of audio and video, including emotion classification accuracy, lower articulation error, and a 15% improvement in confidence estimation correlation. The results demonstrated the advantage of combining multimodal cues for effective communication coaching.

Fields of Study & Subjects

Emotion and Mood Recognition Educational Technology and Pedagogy Intelligent Tutoring Systems and Adaptive Learning

Publication details

Journal / source
International Conference on Computing, Communication, Control and Cyber-Physical Systems (I5CPS)
Pages
1–5
Publication date
26 February 2026
Language
EN