One-Cycle Structured Pruning via Stability-Driven Subnetwork Search

Existing structured pruning typically involves multi-stage training procedures that often demand heavy computation. Pruning at initialization, which aims to address this limitation, reduces training costs but struggles with performance. To address these challenges, we propose an efficient one-cycle structured pruning framework that integrates pre-training, pruning, and fine-tuning into a single training cycle without compromising model performance, referred to as the ‘one-cycle approach’. The core idea is to search for the optimal sub-network during the early stages of network training, guided by norm-based group saliency criteria and structured sparsity regularization. We introduce a novel pruning indicator that identifies the stable pruning epoch by measuring the similarity between evolving pruning sub-networks across consecutive training epochs. Additionally, group sparsity regularization helps accelerate the pruning process, thereby speeding up overall training. Extensive experiments on the CIFAR-10/100 and ImageNet datasets using VGGNet, ResNet, and MobileNet architectures demonstrate that our method achieves state-of-the-art accuracy while being one of the most efficient pruning frameworks in terms of training time. Our code is available at https://github.com/ghimiredhikura/OCSPruner.