ICSD: An Improved Approach for Clean-Sample-Guided Self-Knowledge Distillation With Data Augmentation

Data augmentation (DA) is widely used to improve the generalization of deep neural networks. While its integration with traditional two-stage Knowledge Distillation (KD) has been extensively explored, combining DA with online Self-Knowledge Distillation (SD) often leads to suboptimal performance due to the absence of a reliable teacher model. To address this limitation, we propose the Improved Clean-Sample-Guided Self-Distillation (ICSD) framework. Unlike previous SD approaches that generate unstable or noisy teacher predictions under strong augmentations, ICSD leverages clean samples as teachers to provide consistent and semantically aligned supervision for their augmented counterparts. The proposed method is compatible with both label-preserving and label-mixing augmentations and can be extended to generative augmentation strategies. Extensive experiments on CIFAR-10, CIFAR-100, CINIC-10, and Tiny ImageNet-200 datasets demonstrate that ICSD reduces the top-1 classification error by 1.0–1.6% compared with existing SD methods. Moreover, integrating ICSD with GAN-based augmentation yields an additional 0.2–0.5% improvement on more challenging datasets. These results confirm that ICSD enhances teacher-student consistency and improves robustness against diverse augmentation perturbations.