A Multimodal Audiovisual Deep Learning Framework for Early Detection of Parkinson’s Disease

AbstractParkinson's disease (PD) is a progressive neurodegenerative disorder primarily caused by the degeneration of dopamine-producing neurons in the substantia nigra, leading to typical motor symptoms such as tremors, rigidity, and bradykinesia, as well as non-motor manifestations including depression, sleep disturbances, and speech impairments. Among these, speech abnormalities occur in approximately 90% of patients, making acoustic analysis a promising non-invasive indicator for early detection. However, subtle speech variations are often imperceptible to the human ear, and relying solely on speech signals may overlook complementary visual cues such as hypomimia—reduced facial expressivity commonly observed in PD patients.To address this limitation, we propose PDAF-Net, a novel multimodal deep learning framework that integrates acoustic and facial dynamic features for early PD detection. The architecture comprises an audio branch (1D-CNN + BiLSTM) and a visual branch (2D-CNN + Transformer), with a cross-modal attention mechanism and an iterative attentional feature fusion (iAFF) module designed to enhance bidirectional interaction and modality-specific feature alignment. In addition, a class-balanced focal loss is employed to mitigate sample imbalance introduced by sliding window segmentation.Experiments on a self-collected Chinese multimodal dataset involving 100 PD patients and 100 healthy controls demonstrate that PDAF-Net significantly outperforms unimodal baselines across multiple metrics, achieving 89.3% accuracy and 0.884 F1-score. These findings underscore the effectiveness of multimodal fusion for improving early and automated PD screening in real-world clinical scenarios.