Visual-audio-based fusion network via enhanced transformer for depression detection

Depression, as a common psychological disorder, poses a potential threat to public safety in society. Although many approaches have been proposed to address the long tail distribution issue, these methods still face challenges in modeling long-term dependencies and feature selection. To address these issues, this paper proposes a visual-audio fusion network framework via an enhanced transformer. Concisely, a learnable Multimodal Alignment Module (MAM) is designed to uniformly map video and audio features to a consistent spatiotemporal resolution. Then, a bidirectional Crossmodal Interaction Module (CIM) is introduced to enable video and audio to query/context to each other, achieving fine-grained and symmetrical semantic acoustic coupling modeling. Finally, we design an Enhanced Transformer Module (ETM), which combines a randomly deep, regularized Transformer backbone with dynamic absolute position encoding, thereby improving generalization and adaptability to variable-length inputs in small-sample scenarios while enhancing the ability to model long-term dependencies. Extensive quantitative experiments on public datasets show that our method achieves higher classification accuracy and precision in depression classification than existing methods.