COMPACT: Common-token Optimized Model Pruning Across Channels and Tokens

Improving the memory efficiency, throughput, and serving cost of large language models (LLMs) is critical for edge deployment, interactive applications, and sustainable inference. Pruning is a promising approach, but existing methods have limitations: width pruning disrupts the standard transformer architecture and requires custom inference code, while depth pruning causes abrupt accuracy drops. Moreover, many approaches that work well for LLMs fail to preserve performance on small language models (SLMs). We propose COMPACT, which jointly prunes (i) rare vocabulary to shrink embedding layers and (ii) FFN intermediate channels using common-token–weighted activations aligned with the post-pruning token distribution. COMPACT inherits strengths of both depth and width pruning, such as deployment-friendliness, scale-adaptivity, competitive pruning speed, and strong inference performance. Experiments on several LLM families (0.5B–70B) show state-of-the-art downstream performance, with substantial improvements in inference throughput and GPU memory. Project code is at https://github.com/ekwek1/COMPACT.