Model Minification: A Taylor-Ridge Framework for Structured Compression

As Large Language Models (LLMs) scale, the deployment cost on commodity hardware becomesprohibitive. While unstructured pruning offers theoretical compression, it often requires specializedkernels to realize speedups. We propose a robust Structured Minification framework that physicallyreduces the intermediate dimensions of Transformer MLPs, ensuring compatibility with standard GEMMoperations. Our methodology combines (1) a global Taylor-First-Order sensitivity analysis to identifyredundant feature dimensions, and (2) a closed-form Ridge Regression reconstruction to optimallyheal the output distribution of the pruned layers.We investigate the efficacy of this approach across model scales, applying it to a parameter-dense135M model and a 1.7B model. Our results demonstrate that minification is highly effective even forsmaller, dense models at high retention rates: the 135M model retains significant coherence at 90%retention (Perplexity 4.33 → 4.89). Furthermore, we observe a strong scaling law: the 1.7B model exhibitsremarkable robustness, tolerating 30% structural removal with only minor degradation (Perplexity 3.16→ 4.09). This suggests that while smaller models require conservative minification (80-90% retention),larger over-parameterized models possess a highly compressible subspace recoverable via linear least-squares.Furthermore, because our framework reduces model topology without altering weight precision, itremains strictly orthogonal to quantization, enabling composite compression pipelines that leverage bothstructural minification and subsequent bit-width reduction.The code is available at https://github.com/VladimerKhasia/minisp