Leveraging Multimodal Deep Learning to Identify Trait Anxiety: Algorithm Development and Validation Study (Preprint)

BACKGROUND Trait anxiety is a stable personality characteristic that plays a critical role in mental health and vulnerability to anxiety-related disorders. Its assessment relies primarily on self-report questionnaires, which are susceptible to social desirability and reporting bias. Advances in artificial intelligence and multimodal behavioral analysis provide opportunities to develop more objective and non-invasive assessment approaches. OBJECTIVE This study aimed to develop and evaluate an automated deep learning–based framework for detecting trait anxiety using multimodal behavioral data, including facial expressions, speech, and language. METHODS Trait anxiety was assessed using the State–Trait Anxiety Inventory (STAI) administered during structured cognitive interviews, enabling synchronous collection of self-report and behavioral data. Participants were categorized into high and low trait anxiety groups based on STAI scores. The resulting dataset comprised 46,934 facial images, 380 speech recordings, and corresponding textual transcripts derived from interview responses. Modality-specific deep learning models were developed: an S3D network for facial expression analysis, a ResNeXt-18 network for speech-based acoustic features, and a BERT model for semantic language analysis. Model performance was evaluated using 5-fold cross-validation. A weighted decision-level fusion strategy was applied to integrate outputs from unimodal models into a multimodal ensemble. Classification performance was assessed using accuracy, precision, recall, and F1 score. RESULTS Among the unimodal models, the language-based model achieved the highest accuracy (88.35%), followed by the speech model (77.97%) and the facial expression model (70.07%). The multimodal fusion model further improved performance, achieving an overall accuracy of 89.87%, with superior precision, recall, and F1 scores compared to any single modality. These findings indicate that semantic information contributed most strongly to trait anxiety classification, while multimodal integration provided incremental predictive gains. CONCLUSIONS Trait anxiety can be effectively identified using multimodal behavioral signals derived from structured interviews. The proposed framework demonstrates the advantages of integrating facial, vocal, and linguistic features and offers a non-invasive, objective methodological tool for research-oriented trait anxiety assessment. This approach may support scalable applications in digital mental health contexts, including online psychological counseling services.