The Pitfalls of KV Cache Compression

KV cache compression promises increased throughput and efficiency with negligible loss in performance.While the gains in throughput are indisputable and recent literature has indeed shown minimal degradation on particular benchmarks, in general the consequences of compression in realistic scenarios such as multiinstruction prompting have been insufficiently studied.In this paper, we identify several pitfalls that practitioners should be aware of when deploying KV cache compressed LLMs.We evaluate five KV cache compression methods (StreamingLLM, SnapKV, TOVA, H2O, and K-Norm) on Llama3 8B and Qwen2.5 14B under multi-instruction prompting with IFEval.Importantly, we show that certain instructions degrade much more rapidly with compression, effectively causing them to be completely ignored by the LLM.As a practical example, we highlight system prompt leakage as a case study, empirically demonstrating the impact of compression on leakage and general instruction-following.We identify several factors that contribute to system prompt leakage: compression method, instruction order, and KV eviction bias.We then propose simple changes to KV cache eviction policies that can reduce the impact of these factors and improve the overall performance in multi-instruction tasks.alexluchen/pitfalls-of-kv-cache-compression