Enabling the Use of Approximate Memories in Quantized Large Language Models (LLMs): Dealing with Errors on the Scaling Factors

The use of Large Language Models (LLMs) poses significant challenges in terms of energy dissipation and memory requirements. A widely used approach to reduce memory usage is to quantize the LLMs' parameters, i.e. using a small number of bits. Hence, groups of weights are scaled by a factor and then quantized with fewer bits. To reduce energy dissipation approximate memories, for example, DRAMs with long refresh periods, can also be utilized; however, these memories may cause errors to appear on some bits that can impact LLMs' performance. The impact of these errors on the quantized weights has been widely studied; however, to the best of the authors' knowledge the impact of errors on the scaling factors has not been analyzed, probably because the number of weights is significantly larger than the scaling factors. However, considering errors on scaling factors is important because such errors usually impact an entire group of weights. This paper evaluates for the first time the dependability of two widely used conversational LLMs (LLaMA2-7B and Mistral-7B) against errors on the scaling factors. The impact of errors when using different quantization parameters such as bit width and group size are considered. The results show that: 1) errors on the 1 st exponent bit of the scaling factors of some weights cause severe performance degradation; 2) the impact of errors is larger when using smaller bit widths. The analysis and the simulation-based results are utilized to propose and assess an efficient scheme for the protection of scaling factors; this scheme enables the use of approximate memories with error rates as high as 10-2 .