A Five-Year Journey to Accelerate Homomorphic Encryption With GPUs, Demonstrated by Sub-25-ms Convolutional Neural Network Inference

Driven by growing interest in homomorphic encryption (HE)—a nextgeneration encryption technology that enables direct computation on encrypted data—the DARPA DPRIVE program launched in early 2021 with a goal of developing an ASIC capable of inferring a seven-layer convolutional neural network (CNN7) on encrypted data under 25 ms within a four-year program, a target that seemed daunting at the time. In 2025, we achieved this goal using an off-the-shelf GPU. This paper presents our journey to this remarkable milestone.We synergistically leverage a state-of-the-art GPU implementation of the CKKS HE scheme, a method to replace costly homomorphic ReLU with quadratic functions refined through knowledge distillation, and an optimized convolution algorithm that minimizes expensive homomorphic rotations. Further, we introduce a new convolution method that further reduces rotation overhead in deeper CNN layers with smaller feature maps. Together, these efforts enable CNN7 inference in just 22.4 ms on an NVIDIA RTX 5090 GPU.