Dependable Operation of Large Language Models against Soft Memory Errors: The Quantized Key-Value (KV) Cache Scenario

The Transformer architecture has caused the widespread development of large language models (LLMs) for improving performance in a wide range of natural language processing tasks. One of the fundamental innovations of Transformers is the new mechanism referred to as Attention. Attention can capture complex relations among words in a text, but it requires significant computation; to address this issue, most LLMs implement a cache that stores the Keys and Values (known as KV cache) to avoid re-computation. This cache can be of significant size, sometimes even larger than for the LLM parameters and thus, it is critical for dependable operation, such as for example in the presence of soft memory errors. The impact of soft errors on the memory for KV cache has not been studied, so this paper considers errors on 8-bit and 4-bit quantized KV caches of a widely used LLM, LLaMA3 3B; an evaluation is pursued by fault injection, and the following valuable conclusions are drawn: 1) the Key cache is more robust to errors than the Value cache; 2) the two most significant bits (MSBs) are critical for the dependable operation of the KV cache; 3) errors increasing the amplitude of the values cause a larger performance loss. Based on these findings, we propose a selective protection scheme for the two MSBs; the proposed protection scheme for the KV cache is then integrated with those for embeddings and weights, to show that all the three memory components can be simultaneously protected.