Iterative Layerwise Distillation for Efficient Compression of Large Language Models

Abstract In this paper, distillation methods for large language models with the aim of developing compact models that maintain high performance have been studied. Several existing approaches have been reviewed and their strengths and weaknesses have been discussed. In the study, an improved method based on the ShortGPT approach and using the idea of iteratively evaluating the importance of layers has been developed. At each stage, importance has been estimated by measuring the performance degradation when removing individual layers using representative datasets. This process has been combined with further training using a combined loss function based on the Kullback–Leibler divergence and mean squared error. Experiments with the Qwen2.5-3B model have shown that the number of layers can be reduced from 36 to 28 (resulting in a model with 2.47 billion parameters) with a quality reduction of only 9.7%, and to 24 layers with an 18% quality reduction. The results have indicated that the middle layers of the Transformer contribute less to the inference process, which highlights the potential of the proposed method for creating efficient models. The results have also confirmed the effectiveness of iterative distillation and subsequent fine-tuning, making this approach suitable for use in computationally constrained environments.