Fast and Secure Selective Homomorphic Encryption for Federated Learning

Federated learning (FL) enables collaborative model training among data providers without sharing raw data. However, model updates remain vulnerable to attacks such as model inversion and gradient leakage. Fully homomorphic encryption (FHE) ensures strong confidentiality, but can incur prohibitive computational and communication overhead, while differential privacy (DP) may degrade model accuracy. We propose a hybrid framework that selectively applies homomorphic encryption (HE) to a subset of model parameters, while protecting the remaining parameters with DP noise and lightweight bitwise scrambling. This design preserves privacy guaranties while reducing overhead by nearly an order of magnitude compared to FHE. Experiments on CIFAR-100 and a medical imaging dataset show that our framework achieves security similar to FHE while drastically lowering cryptographic cost. For example, on CIFAR-100 with MobileNetV2, fully homomorphic encryption required 68 minutes per round, whereas FAS reduced the time to 18 minutes (with an overhead of 6 minutes compared to 56 minutes for FHE), representing almost a tenfold reduction in overhead while maintaining model accuracy above 90%. Our framework was up to 2× faster than state-of-the-art techniques, namely MaskCrypt and FedML-HE, and achieved comparable security.