BranchySplit: Runtime-Adaptable Partitioning and Early Exits for Accelerated Edge Inference

Deploying deep neural networks (DNNs) in real-time, resource-constrained edge environments remains a significant challenge due to the increasing depth and computational demand of state-of-the-art models. While state-of-the-art split computing and early-exit architectures independently offer solutions to reduce inference latency and offloading overhead, they are rarely integrated in a unified framework. This paper introducesBranchySplit, a novel architecture that combines dynamic state-of-the-art split computing with early exit mechanisms to accelerate inference in edge-cloud systems. Our method employs a shared dynamic autoencoder to support multiple partitioning points, enabling runtime adaptability based on varying system conditions. By integrating early exit branches at intermediate layers, the model can terminate inference early when confidence is sufficient, further reducing computation and communication. Extensive experiments on Caltech-256 and Tiny-ImageNet dataset using four backbone models (VGG-16, ResNet-50, Efficientformerv2-S2, and ConvNeXT-small) show that our approach consistently outperforms the traditional split computing approaches, achieving up to 2.7xlatency speed-up compared to an all-in-server solution, while maintaining comparable or even improved accuracy. In addition, our proposed dynamic compression pipeline achieves a higher speed-up compared to the pipeline in similar work, i.e. SPINN, with up to 10xfor ResNet50 on caltech-256. These results demonstrate the efficacy of BranchySplit in enabling low-latency, resource-efficient vision inference in heterogeneous edge-cloud settings.