DIVINE : Coordinating Multimodal Disentangled Representations for Oro-Facial Neurological Disorder Assessment

Read full text Summarise with VIRAT
Authors: Mohd Mujtaba Akhtar, Girish, Muskaan Singh
Cited by: 0
References: 24

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

In this study, we present a multimodal framework for predicting neuro-facial disorders by capturing both vocal and facial cues.We hypothesize that explicitly disentangling shared and modality-specific representations within multimodal foundation model embeddings can enhance clinical interpretability and generalization.To validate this hypothesis, we propose DIVINE a fully disentangled multimodal framework that operates on representations extracted from state-of-the-art (SOTA) audio and video foundation models, incorporating hierarchical variational bottlenecks, sparse gated fusion, and learnable symptom tokens.DIVINE operates in a multitask learning setup to jointly predict diagnostic categories (Healthy Control, ALS, Stroke) and severity levels (Mild, Moderate, Severe).The model is trained using synchronized audio and video inputs and evaluated on the Toronto NeuroFace dataset under full (audio-video) as well as single-modality (audioonly and video-only) test conditions.Our proposed approach, DIVINE achieves SOTA result, with the DeepSeek-VL2 and TRILLsson combination reaching 98.26% accuracy and 97.51% F1-score.Under modality-constrained scenarios, the framework performs well, showing strong generalization when tested with video-only or audio-only inputs.It consistently yields superior performance compared to unimodal models and baseline fusion techniques.To the best of our knowledge, DIVINE is the first fully disentangled multimodal framework to jointly perform categorical diagnosis and severity estimation for oro-facial neurological disorders using synchronized speech and facial video.

Fields of Study & Subjects

Multimodal Machine Learning Applications Emotion and Mood Recognition Voice and Speech Disorders

Publication details

Journal / source
Conference of the European Chapter of the Association for Computational Linguistics
Pages
5379–5392
Publication date
1 January 2026
Language
EN