Structure-aware knowledge distillation using centered kernel alignment

Knowledge distillation (KD) is a powerful technique for model compression, but conventional feature-based methods often overlook global structural consistency. We propose Structure-Aware Knowledge Distillation (Str- KD), which explicitly enforces structural correspondence between teacher and student models. By leveraging Centered Kernel Alignment (CKA), Str-KD partitions teacher layers into functional groups based on representational similarity. The student layers are then divided into the same number of groups and aligned with the teacher’s structure to ensure holistic consistency. A novel CKA-based loss then encourages the student to mimic the teacher’s hierarchical processing. Experiments on CIFAR-100 using VGG and ResNet architectures demonstrate that Str-KD consistently outperforms Vanilla KD and other feature-based baselines. Our results show that incorporating global structural information is crucial for effective and efficient knowledge transfer.