Boosting Knowledge Distillation Via Local Categories Similarity Scaling

Knowledge distillation (KD) transfers knowledge from a pre-trained teacher network to a lightweight student model. While most KD methods adjust global category similarities by softening the teacher’s output distribution, they struggle to regulate similarities among local categories. This often leads to overconfident predictions and reduced discriminability for similar classes. To address this, we propose Local Categories Similarity Scaling (LCSKD), a plug-and-play framework that enhances KD by refining local similarity transfer. Specifically, we partition the predicted category probabilities into local groups and distill intra-group relations. A temperature adjustment strategy based on distribution "smoothness" is further introduced to calibrate local similarities effectively. LCSKD introduces negligible computational overhead and is compatible with existing KD methods. Experiments on visual tasks validate its effectiveness.