Exploring the Dynamic Interplay of Cognitive Load and Emotional Arousal by Using Multimodal Measurements: Correlation of Pupil Diameter and Emotional Arousal in Emotionally Engaging Tasks

Multimodal data analysis and validation based on streams from state-of-the-art sensor technology, such as eye-tracking or emotion recognition using the Facial Action Coding System (FACS) with deep learning, allows educational researchers to study multifaceted learning and problem-solving processes and to improve educational experiences. This study aims to investigate the correlation between two continuous sensor streams—pupil diameter as an indicator of cognitive workload and FACS with deep learning as an indicator of emotional arousal (RQ1a)—specifically for epochs of high, medium, and low arousal (RQ1b). Furthermore, the time lag between emotional arousal and pupil diameter data will be analyzed (RQ2). A total of 28 participants worked on three cognitively demanding and emotionally engaging everyday moral dilemmas while eye-tracking and emotion recognition data were collected. The data were preprocessed in Phyton (synchronization, blink control, and downsampling) and analyzed using correlation analysis and Granger causality tests. The results show negative and statistically significant correlations between the data streams for emotional arousal and pupil diameter. However, the correlation is negative and significant only for epochs of high arousal, while positive but nonsignificant relationships were found for epochs of medium or low arousal. The average time lag for the relationship between arousal and pupil diameter was 2.8 ms. In contrast to previous findings without a multimodal approach suggesting a positive correlation between the constructs, the results contribute to the state of research by highlighting the importance of multimodal data validation and convergent validity. Future research should consider emotional regulation strategies and emotional valence.