MMFC-PFL: Multimodal Feature Collaborative Personalized Federated Learning

Federated learning (FL) aims to improve model performance while protecting user privacy. The amalgamation of multimodal data, encompassing diverse data types, introduces challenges in FL scenarios. To address these challenges, we propose an innovative multimodal feature collaborative personalized federated learning framework (MMFC-PFL) grounded in the unique characteristics of multimodal features. It includes multimodal collaborative loss and personalized parameter aggregation. The multimodal collaborative loss comprises intra-modal loss, aimed at proficiently representing unimodal features, and inter-modal loss, designed to seamlessly integrate collaborative global multimodal features. The process of personalized parameter aggregation entails the discerning selection of clients during initialization in each global round, with a specific emphasis on those that exhibit maximal separation from the current client for aggregation purposes. The distance between clients is facilitated through Canonical Correlation Analysis on inter-modal features. Additionally, we present an iterative client-wise personalized aggregation strategy that enhances the generalization capabilities of the personalized model. We validate the proposed MMFC-PFL on three datasets: AV-MNIST, MM-IMDB, and MIMIC III. The experimental results demonstrate the effective improvement of model performance and robustness by the proposed multimodal federated learning framework.