Mapping strategy of quantized KV cache of LLMs into mixture of SLC and MLC ReRAM among various quantization methods

Abstract Quantized key-value (KV) cache is a remarkable technology to reduce the memory footprint during large language model (LLM) inference. While multi-level cell (MLC) resistive random-access memory (ReRAM) offers higher density than single-level cell (SLC) ReRAM, its current fluctuation, often induced by random telegraph noise, leads to false detection of cell state and degraded reliability. This work explores a strategy to store quantized KV cache in ReRAM to reduce memory area, while maintaining LLM performance. This study analyzes the error robustness of the three constituent components of the quantized KV cache. The proposal splits the value of each component into the upper bits and the lower bits, then stores the upper bits in SLC ReRAM and the lower bits in MLC ReRAM. This work achieves 49.1% fewer memory cells than the conventional method while maintaining the perplexity at a low level, regardless of LLMs and quantization methods.