Natural Selection of LLMs via Pruning (Student Abstract)

Large Language Models (LLMs) have attained state-of-the- art performance on various tasks. But since most of these models contain billions of parameters, they consume enor- mous amounts of memory, computation and energy re- sources. Thus, they constitute immense challenges in their deployment, inference and training. Conventional pruning methods, while reducing the parameter count effectively, at times make sub-optimal trade offs between efficiency of a model and its performance. In this work, we employ a hybrid approach of LLM-specific pruning strategies and the frame- work that employs Genetic Algorithms (GA) framework for global optimization of pruning patterns. Inspired by biologi- cal evolution. The Pruned Model is then represented as chro- mosomes and is then successively optimized using the opera- tions of selection, crossover, and mutation. This approach, in- spired by the biological evolution process of natural selection, enables a systematic exploration of the pruning space. Our code is available at https://github.com/ankur777jinn/Genetic- Pruning.