FedRSL: Representation Subspace Learning in Model-Heterogeneous Federated Learning

Model-heterogeneous federated learning (MHFL), supporting FL collaboration across clients with heterogeneous models, has become a more practical FL paradigm. Existing MHFL methods enable knowledge fusion over heterogeneous client models by sharing partial homogeneous parameters or extracted label-wise average representations, suffering from model performance bottlenecks and privacy leakage risks. To bridge this gap, we propose a novel model-heterogeneous Federated learning method with homogeneous Representation Subspace Learning (FedRSL) instead of sharing model parameters or representations. In FedRSL, each client’s local heterogeneous model comprises a feature extractor and a prediction header. (1) We construct a homogeneous representation subspace for each client to learn local representation knowledge, and the server aggregates them to generate the global representation subspace for representation knowledge fusion. (2) To facilitate representation learning capability while maintaining efficient communication and computation, we design a lightweight linear model as the homogeneous low-rank linear representation subspace. For each local data sample, its personalized representation extracted by the feature extractor is processed by the global representation subspace to produce the corresponding generalized representation. (3) To effectively bi-transfer global generalized and local personalized knowledge, we reduce the distance between the local personalized representation and the corresponding generalized representation. Experiments on 3 computer vision and 1 natural language processing benchmark datasets over 6 baselines demonstrate that FedRSL obtains state-of-the-art model accuracy (up to 5.51% accuracy improvement) while consuming low communication and computation overheads.