FedKD: A Fine-Grained Parameter-Efficient Federated Co-Tuning Framework With Knowledge Decoupling for Large and Small Foundation Models

Recently, federated foundation models (FedFM) have emerged as a promising approach. FedFM preserves data privacy while co-tuning foundation models (FM) across edge nodes. However, the high complexity and numerous parameters of FM hinder the deployment across diverse resource-constrained edge nodes in the real world. This limitation further leads to the inevitable model heterogeneity problem. In addition, performance bottlenecks and inefficient aggregation caused by inevitable data heterogeneity also limit the deployment of FedFM. In this paper, we propose a fine-grained parameter-efficient federated co-tuning framework based on knowledge decoupling, named FedKD, to simultaneously mitigate the aforementioned issues. It integrates a compact personalized tensor low-rank adapter (PTLoRA) and an efficient two-stage distillation scheme (KDMD). Specifically, PTLoRA explicitly decouples the low-rank weight update tensor of FM and adopts the layer-sharing scheme to enable efficient fine-tuning for enhanced edge models. It enables efficient fine-tuning by tackling inconsistent task-relevant knowledge across layers caused by data heterogeneity. Furthermore, KDMD implicitly decouples the knowledge learned from private datasets and transfers the decoupled general knowledge to the cloud FM. It alleviates the model divergence during aggregation, thereby improving the generalization of the cloud FM. Extensive simulations in various heterogeneous scenarios demonstrate that FedKD improves the generalization of the cloud model and the personalization of edge models. These results highlight the effectiveness of FedKD for supporting various cloud-edge collaborations. Code is available online athttps://github.com/Aprilmin/FedKD.