MPCPFL: Prototype-Based Multimodal Personalized Federated Learning With Missing Modalities in Medical Domain

With the global aging population on the rise, the amount of disease data related to the elderly is also increasing. Effectively protecting personal data when analyzing and processing disease data has become crucial. Federated learning (FL), as a proven method, is widely used. In the multimodal FL for disease diagnosis, the intricate challenge of modality missing and heterogeneous data distribution across healthcare clients severely degrades model performance and generalization. Existing frameworks struggle to address these issues in scenarios with complex modality missing and data heterogeneity, particularly in privacy-sensitive medical applications. In this article, we introduce a novel framework,modal prototype and contrast for personalized federated learning, which enhances prototype-based multimodal federated learning with personalized local aggregation and enables the adaptive fusion of global prototypes and local client models during training. By leveraging the modal prototype to compensate for missing modalities and modal contrastive training to align client representations with shared prototypes, the framework mitigates global model degradation caused by modality absence. In addition, we adapt the SLAKE medical multimodal dataset into a VQAv2-style classification task, simulating realistic disease diagnosis scenarios with varying modality missing rates (10$\sim$90%). Experimental results demonstrate that our approach achieves state-of-the-art performance. Compared to the baselines, our method improved inference accuracy by up to 13.7% when 90% of the modality was missing during training.