Automatic Depression Level Prediction Based on Facial Video With Emotional Cues Using a Hybrid Machine Learning Framework

Depression is a kind of mood disorder. Nevertheless, there is a lack of research on emotional information in the identification of depression. This study intends to explore the role of emotions in the identification of depression by analyzing the performance in facial videos induced by different emotional stimuli. It also aims to address the deficiency of emotional representation in existing video datasets. In order to study the influence of emotions on the prediction of depression levels, we initially collected facial videos from 80 depressed patients and 80 normal subjects while they were performing tasks of watching calm, sad, and joyful video clips, and then constructed an emotional facial video (eFV) dataset with emotional cues. A hybrid framework integrating deep learning and ensemble learning was established for predicting depression levels. This framework utilizes attention-based 3-D convolutional neural network (3-DCNN) to automatically learn and extract the spatio-temporal features of facial videos, assigns significance to different emotions via the emotion-cross-attention mechanism, and finally employs the ensemble learning method (XGBoost) for depression level prediction. Comparative experiments were carried out on the eFV dataset and the widely-used AVEC2013 and AVEC2014 public facial video datasets. The results show that facial videos with emotional cues significantly improve the performance of depression recognition. The evaluation errors (MAE/RMSE) of our proposed model on AVEC2013 and AVEC2014 facial videos are 7.21/9.25 and 6.57/8.36, respectively. However, the evaluation errors on facial videos with sad and happy emotions are 5.06/6.35 and 5.17/6.58, respectively. Furthermore, the fusion of sad and happy emotions achieved the lowest evaluation error of 4.45/5.75. This highlights the critical role of emotional cues in depression recognition.