CoreKD: A Context-Aware Local Region Structural Contrastive Knowledge Distillation Framework for Object Detection

Knowledge distillation (KD) aims to transfer knowledge from a cumbersome teacher to a lightweight student, thereby reducing overall model complexity without sacrificing performance. Current methods tend to focus excessively on pixel-level knowledge transfer while overlooking localized and contextual information. To address this, we propose a novel context-aware local region structural contrastive knowledge distillation framework (CoreKD) for object detection tasks. Specifically, we introduce a patch-based semantic structural distillation (PSD) method, facilitating efficient localized valuable knowledge transfer. This method guides the student model in learning both consistency and diverse local knowledge from the teacher model by integrating the semantic and structural information. We also propose the intra-region contextual (Ita-RC) and inter-region contextual (Ite-RC) constraints for the PSD framework. These constraints could transfer the region-wise contextual information between the student and teacher, significantly enhancing the student detector's capacity to model contextual dependencies. To validate the efficacy of CoreKD, we conduct extensive experiments on the public MS COCO 2017 and PASCAL VOC datasets. These results demonstrate that CoreKD markedly improves the performance of students in object detection tasks, indicating its ability to transfer valuable knowledge.