Extracting Meaningful Insights from User Research Videos

Recognising and tracking user emotions in research videos is vital to understanding user needs and expectations. Limited research exists on automating emotion extraction from multimodal videos in user experience (UX). This study proposes a conceptual framework for automated extraction of actionable insights using facial, speech-to-text, and text-based emotion recognition to capture nuanced emotional data. The multimodal approach integrates visible and spoken cues through temporal alignment and fusion techniques, enabling robust behavioural pattern detection. An interactive AI analyst tool is used to query the integrated data in natural language, reduce manual workload, and improve the efficiency and scalability of UX evaluation. A case study of the implementation of the proposed framework is also provided with details of individual components, such as facial emotion recognition, speech-to-text, text-based emotion recognition, temporal alignment and fusion, and insight extraction via interactive AI.