Compressing Kv Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

The rapid expansion of context windows in Large Language Models (LLMs) enables complex long-document tasks, but dramatically increases inference memory usage due to linear KV cache growth. Existing KV cache compression methods often discard less relevant tokens, risking performance loss when important context is removed. We propose POD (Proximal tokens over Distant tokens), a novel framework that allocates memory based on token importance: proximal tokens are fully preserved, while distant tokens are retained in a compact, shared form rather than discarded. Our approach is based on two key observations: (1) tokens at the beginning and end of the context are most critical for next-token prediction, and (2) attention scores for distant tokens are highly redundant across layers. By sharing attention scores for distant tokens across layers, POD substantially reduces KV cache memory while maintaining essential context. A lightweight post-training adaptation enables the model to adjust to this structure. Experiments on real-world long-context benchmarks show that POD reduces KV cache memory by up to 35% without sacrificing performance. In addition, our method is complementary to token-selection techniques and can be combined for further compression.