FedMIF: Fast-Generalization Image Fusion via Task-Customized Nodes Federated Learning

Image fusion is a common approach to enhancing image quality by combining the strengths of multi-modal visual signals, thereby improving the performance of computer vision tasks. However, privacy concerns regarding raw images leave data from multiple domains and clients isolated, preventing centralized training of a universal image fusion model. Federated learning (FL) is a distributed framework that enables collaborative model training by aggregating parameters from multiple nodes without sharing raw data, offering a privacy-preserving solution for lightweight and generalizable models. In this paper, we propose thefederatedmulti-modalimagefusion (FedMIF), a general multi-modal image fusion solution across multiple domains that operates without raw data sharing. We notice existing FL methods lack task-specific strategies tailored for general fusion tasks, leading to suboptimal fusion performance. To address this, we propose a four-stage framework that enhances detail features while maintaining overall visual quality, which are 1) Server initialization stage, dual decoders are trained on natural data to regulate feature distribution. 2) Task-agnostic stage, all clients collaboratively train the encoder to improve feature extraction performance. 3) Task-specific stage, training is conducted within relevant tasks to address the negative impact of inconsistent optimization objectives across different tasks. 4) Local-specific stage, local adapters are trained on each client to help mitigate detail loss resulting from model aggregation. Complete subjective and objective comparisons prove that FedMIF could generate information-rich fusion results on four fusion tasks while requiring less communication overhead. Our code is available at:https://github.com/suweijian1996/FedMIF.