Enhancing LLM inference via training-free anchor compression: a layer-wise representation projection framework

Large Language Models (LLMs) exhibit substantial redundancy in their intermediate representations, leading to inefficiencies during inference. In this work, we introduce anchor compression, a novel training-free framework designed to accelerate LLM inference by exploiting structural redundancy in hidden activations. Instead of pruning weights or lowering numerical precision, our method projects intermediate representations onto a compact set of anchor vectors, which serve as a low-dimensional basis for subsequent computations. This design preserves the original model parameters, requires no retraining, and provides a tunable mechanism for balancing computational efficiency and task performance. We formulate the theoretical foundation of anchor-based projection and develop an algorithm that integrates seamlessly with standard Transformer architectures. Experiments results establish anchor compression as a lightweight yet powerful approach for efficient large-model inference and open a new perspective on leveraging representational redundancy for model acceleration.