A Survey on Model Compression for Transformer-Based Large Language Models

The mainstreamTransformer-based Large Language Models (LLMs) have demonstrated to exhibit remarkable performance in various Natural Language Processing (NLP) tasks. However, high computational complexity and high memory usage pose significant challenges for deploying LLMs on resource-constrained devices. Model compression has been proven to be an efficient and practical approach to designing efficient LLMs. In this paper, we provide a systematic and comprehensive survey on the methods, evaluation metrics, and benchmarks of model compression for LLMs. Similar to small models, the methods of model compression for LLMs can still be categorized into pruning, quantization, Knowledge Distillation (KD), and low-rank decomposition. Due to the use of key-value (KV) cache to accelerate inference in Transformer-based LLMs, KV Cache is also a key factor in memory consumption when LLMs process long contexts. We summarize the latest methods of KV Cache compression. Besides, we provide a novel perspective on Neural Architecture Search (NAS) for LLMs since NAS is being developed into a novel approach for model compression, whose search stage adopts different efficient search strategies to achieve automated compression of the trained super-nets. We hope our survey will offer valuable perspectives to researchers in the field, thereby inspiring them to participate in research or contribute more valuable suggestions.