SplitFed-CKD: SplitFed Learning With Contrastive Learning and Knowledge Distillation on Non-IID Datasets

High-quality Deep Learning (DL) enabled by large-scale training data has found applications across various industries. However, when local data contains sensitive privacy information, the amount of data to be aggregated is limited, hindering sufficient training. Federated Learning (FL) and Split Learning (SL) enable collaborative training while protecting the privacy information of local data, but each approach faces distinct difficulties, such as computational load and limited parallelism. SplitFed Learning (SFL), a hybrid of SL and FL, provides a promising method to overcome these issues. However, when dealing with non-Independent and Identically Distributed (non-IID) datasets, a common scenario in real-world applications, SFL performance suffers considerably as a result of client drift and biased model updates. To address this problem, this paper proposes SplitFed-CKD, which integrates contrastive learning and knowledge distillation into SFL through the use of an auxiliary classifier that takes the outputs of client models as input. Specifically, SplitFed-CKD classifies the output of the global model into reliable positive and unreliable negative samples according to dynamic thresholds. Contrastive learning is then performed on the feature vectors of the auxiliary classifier using prototypes that represent averaged samples. This enables each client-side model to effectively learn global knowledge and mitigate biased training. Furthermore, knowledge distillation is performed by using the training model as a teacher and the client and auxiliary classifiers as students. Finally, comparisons with recent state-of-the-art approaches, together with five-run statistical significance tests, confirm that SplitFed-CKD achieves consistently higher accuracy with statistically significant gains in large-class non-IID settings where class diversity is high.