Hyperion: Private Token Sampling with Homomorphic Encryption
Abstract
A promising direction for enabling private queries to large language models (LLMs) is with homomorphic encryption (HE).An open problem is performing token sampling under HE.In this paper, we introduce Hyperion, an efficient HE algorithm for inverse transform sampling, enabling private token sampling with 1 comparison depth, O(1) amortized comparisons, and O(log n) rotations.We implement our approach and demonstrate that it samples tokens in 0.14 seconds for 32k tokens (≈ 4.4 µs per token) on GPU, achieving a 100× latency improvement over prior work.
Fields of Study & Subjects
Cryptography and Data Security
Privacy-Preserving Technologies in Data
Advanced Steganography and Watermarking Techniques
Publication details
- Journal / source
- Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Publisher
- Association for Computational Linguistics
- Pages
- 14150–14159
- Publication date
- 1 January 2026