Implementation and Evaluation of Multi-Hop Parallel Split Learning

Offloading techniques like Split Federated Learning (SplitFed) or Parallel Split Learning (Parallel SL) enable multiple resource-constraineddata ownerdevices to participate in collaborative training processes with the help of resourcefulcompute nodes. Recent findings indicate that the number of compute nodes (hops) significantly impacts the training delay. Yet, determining the ideal number of hops is not an easy task. Therefore, in this work, we propose a mathematical model that estimates the training delay of single- and multi-hop Parallel SL. This tool not only helps in determining the optimal number of hops before deployment, but also serves as an evaluation tool in future research works. Further, we construct a lightweight optimization problem that targets maximizing the pipeline parallelism at a theoretical level. Also, we present the SplitPipe framework, which allows the support of pipeline parallelism at the system level as well. Finally, we conduct a thorough numerical evaluation, which first validates the accuracy of the proposed estimation model and then presents a detailed analysis of multi- and single-hop Parallel SL.