Asymmetric KV Cache Compression Using State-Aware Sparsity and Quantization

To enable the efficient deployment of Large Language Models (LLMs) on resource-constrained devices, recent studies have explored Key-Value (KV) Cache compression, such as quantization and sparsity-based methods. However, prior works suffer from limitations in both accuracy and memory efficiency, as they overlook the structural differences between Key and Value states and the interaction between sparsity-based compression and quantization. Moreover, previous works often underexplored the variability of generated token lengths across requests, leading to inconsistent performance.To address these limitations, we present a new asymmetric, state-aware KV Cache compression pipeline. The proposed pipeline tailors quantization granularity, sparsity-based compression type, and sparsity level to each state guided by our analysis of methodological characteristics and state-specific structural properties. Furthermore, we demonstrate that the accuracy degradation and KV Cache footprint under compression are sensitive to the ordering of different operations, highlighting the need for an order-aware design. Building on our state- and order-specific insights, we assess the influence of individual channels before applying Rotary Positional Embedding (RoPE), and define a new criterion for structured channel pruning, enabling pre-RoPE pruning to mitigate accuracy loss. We also incorporate parameter-reusing techniques and avoid extra decoding-time computation and memory overhead. Experimental results demonstrate that our approach achieves superior accuracy-memory trade-offs and consistent memory savings across sequence lengths in both quantized and non-quantized settings.