JanusQuant: Accurate and Efficient 2-bit KV Cache Quantization for Long-Context Inference
Abstract
Long-context large language models (LLMs) have seen widespread adoption in recent years. However, during inference, the key-value (KV) cache—which stores intermediate activations—consumes significant memory, particularly as sequence lengths grow. Quantization offers a promising path to compress KV cache, but existing 2-bit approaches fall short of achieving optimal inference efficiency due to hardware-unfriendly algorithms and system implementations.
Fields of Study & Subjects
Advanced Neural Network Applications
Topic Modeling
Low-power high-performance VLSI design
Publication details
- Journal / source
- ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming
- Pages
- 301–314
- Publication date
- 28 January 2026