EEG- and Micro-Expression-Based Emotion Recognition and Consciousness Detection

In the field of Disorders of Consciousness (DoC), traditional diagnostic methods often struggle with high misdiagnosis rates and lack the sensitivity needed to discern subtle emotional responses, necessitating more accurate and comprehensive assessment tools. Current emotion recognition techniques, primarily using single-modality data such as electroencephalography (EEG) or facial expressions, fail to capture the complex interplay of emotional states in patients with DoC, leading to incomplete assessments. To address these challenges, this study proposes a novel multimodal fusion framework that integrates heterogeneous EEG and micro-expression data through an innovative spatio-temporal attention mechanism. This approach significantly enhances the detection and classification of emotional states by leveraging differential entropy features from EEG and temporal features from micro-expressions, fused using deep learning techniques. The proposed method not only achieves mainstream models in terms of accuracy but also demonstrates potential for broad application in medical diagnostics, providing a basis for more personalized patient care and opening avenues for future research in clinical practice settings. We found that two patients had significantly better indicators than the others, indicating better brain activity levels and outcomes in these two patients, which is also consistent with the clinical results one month after the experiment. By effectively merging data from multiple sources, our study sets a new benchmark for emotion recognition in DoC, contributing to more accurate clinical assessments and enhancing the quality of life for affected patients.