Heterogeneous Multimodal Federated Learning With Missing Modality via Mask-Restoration and Self-Guidance

Federated learning (FL) is well-suited for multimodal tasks due to its ability to protect privacy and support local training. However, the complexity of real-world sensor environments causes modality heterogeneity across clients. Some modalities may be missing altogether, making it difficult to construct a generalized global model. Existing multimodal federated learning methods often address modality-missing scenarios under simplified assumptions of modality heterogeneity, typically focusing on unimodal clients and modality-complete multimodal clients. Moreover, to mitigate performance degradation caused by missing modalities, some approaches assume the availability of auxiliary information at the server, which may be impractical in real-world scenarios. Therefore, we propose a novel heterogeneous multimodal Federated Learning with Mask-Restoration and Self-Guidance (FL-MRSG). The Mask-Restoration employs a masking strategy to simulate missing data during feature extraction, enabling the network to learn semantic features of missing modality. Furthermore, we introduce an innovative self-guidance mechanism that leverages the restored data as guidance information, enabling the network to distinguish between complete and missing data representations. In addition, we propose a personalized decoupled aggregation strategy to facilitate the collaborative training of a global model across heterogeneous modality clients. We extend the multimodal test set to arbitrary modality combinations to evaluate the robustness of the global model. Extensive experiments on MOSI and SIMS datasets demonstrate the effectiveness of the proposed FL-MRSG for arbitrary missing modalities.