ReLUPruner: Rethinking ReLU Importance with Taylor Expansion for Efficient Private Inference

With the growing adoption of Machine-Learning-As-A-Service (MLaaS), Private Inference (PI) has emerged as a promising solution to address its security concerns through cryptographic techniques. However, nonlinear operations in neural networks contribute the majority of the computational and communication overhead in PI. Existing studies mainly focus on optimizing and reducing the number of ReLU activations in neural networks,but traditional pruning methods may mistakenly remove ReLUs that are critical for maintaining model accuracy. To accurately evaluate the importance of ReLU in the network, we propose ReLUPruner, a method that uses Taylor expansion to accurately quantify the impact on loss before and after ReLU replacement. Furthermore, we establish a hierarchical importance metric to guide layer-wise ReLU budget allocation and adopt a progressive pruning strategy to dynamically adjust the pruning rate of each layer according to the training progress. Extensive experiments on various models and datasets show that ReLUPruner achieves a good balance between ReLU budget and model accuracy, with a 1.89\% improvement over the existing SOTA method 1.89\% (90.12\%, 12.9k ReLU budget, CIFAR10), 3.62\% (77.9\% 50k ReLU budget, CIFAR100) and 2.66\% (56.13\%,30k ReLU budget, Tiny-ImageNet), while achieving superior accuracy even without additional fine-tuning stage.