MPCF: Multi-stage progressive cross-modal fusion for depression severity prediction using audio-Visual modalities

Depression is a widespread mental health disorder, making early and accurate assessment essential for effective intervention. Current clinical methods largely depend on subjective rating scales, which could be significantly improved by incorporating objective measures derived from behavioral analysis. In this paper, we introduce the Multi-stage Progressive Cross-modal Fusion (MPCF) framework for predicting depression severity by leveraging both audio and visual modalities. Traditional unimodal approaches often struggle to capture the complex and multidimensional nature of depression due to their inherent limitations, resulting in a narrow focus. While audio-visual multimodal methods make progress in addressing these challenges, many existing single-stage fusion architectures fall short in facilitating multi-level interactions, leading to suboptimal utilization of cross-modal temporal dynamics and complementary features. The MPCF framework addresses these limitations through three core innovations: (1) an adaptive spatiotemporal attention mechanism that dynamically prioritizes facial regions and temporal segments most relevant to depression; (2) a multi-stage progressive fusion strategy that fosters deep interactions between audio and visual features across various levels of granularity; and (3) enhancements in robustness achieved through pretraining and multimodal data augmentation. Comprehensive experiments conducted within the MPCF framework on the AVEC2013 and AVEC2014 datasets yield state-of-the-art performance, achieving RMSE/MAE scores of 6.41/5.12 and 5.90/4.82, respectively. These results underscore the effectiveness of our approach in accurately assessing depression severity. The proposed MPCF framework holds significant promise as an auxiliary tool for evaluating clinical depression, offering objective behavioral indicators that can complement existing assessment methods and improve clinical outcomes.