SHARP: Structured Hierarchical Attention Rank Projection for Efficient Language Model Distillation

Read full text Summarise with VIRAT
Authors: Jieui Kang, Eunjoung Yoo, Soeun Choi, Yeonhee Kim, Jaehyeong Sim
Cited by: 0
References: 32

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Knowledge distillation has emerged as a crucial technique for compressing large language models into more deployable versions. While existing approaches focus on transferring knowledge at length-based linguistic granularities such as tokens, phrases, and sequences, they often fail to capture the intrinsic hierarchical attention mechanisms of modern transformer architectures. Furthermore, when multiple granularities share a single representation space, they compete for the same capacity, causing complementary information to be lost. We term this phenomenon representation collapse. To address this limitation, we propose SHARP (Structured Hierarchical Attention Rank Projection), a novel distillation framework that decomposes attention patterns into token-level, head-level, and layer-level representations and projects each into a dedicated orthogonal subspace prior to distillation. This mathematical decomposition provides each granularity with independent representational capacity, preventing interference while enabling parallel optimization pathways. Through extensive experiments on both natural language generation (NLG) and natural language understanding (NLU) tasks with teacher models ranging from 350M to 6.7B parameters distilled to a 125M parameter student, SHARP consistently outperforms existing distillation methods. It achieves an average of 5.2% perplexity reduction on NLG tasks relative to the strongest baseline, with gains reaching 7.8% for the 6.7B teacher configuration. On NLU tasks, SHARP shows consistent accuracy improvements of 1.5 to 3.2% across all model scales. These results confirm that orthogonal capacity allocation is an effective principle for multi-granularity knowledge transfer in large language model compression.

Fields of Study & Subjects

Topic Modeling Advanced Neural Network Applications Domain Adaptation and Few-Shot Learning

Publication details

Journal / source
IEEE Access
Volume
14
Pages
57983–57996
Publication date
1 January 2026
Language
EN