Rethinking Knowledge Distillation through a Unified Framework

Current Knowledge Distillation (KD) methods claim their own explainability principle of knowledge. However, these methods lack a unified framework to review the KD process. In this paper, we re-investigate three primary types of knowledge (Response-Based, Feature-Based, and Relation-Based Knowledge) in knowledge distillation in a unified framework. To achieve this, we propose a novel representation of knowledge, consisting of two components: Feature Selection and Feature Transformation. The former is the extent of information on features being selected for transfer in KD. The latter is the transformation controlled with different factors to reshape the features. We strictly prove that the proposed framework contains three primary types of knowledge under a gradient backpropagation perspective. Besides, we further show that different KD methods transfer the same knowledge and serve the same goal of empirically bringing features closer to the student and teacher networks. Extensive experiments demonstrate that both the Feature Selection and Feature Transformation of KD methods are significantly important for its performance. Also, our experiments reveal the nature of the true knowledge in KD to be the prior information of the adjacency relationship among the dataset categories, i.e., non-target class probability in the neural networks.