Progressive Learning Scheduling for Knowledge Distillation

Knowledge Distillation (KD) transfers the knowledge of a large pretrained teacher model into a smaller student model, but the training curriculum—i.e., the schedule and weighting of the distillation signal—remains an open design question. We introduce PLS-KD (Progressive Learning Scheduling for Knowledge Distillation), a family of curriculum-based distillation strategies that modulate how soft targets from a 128-hour teacher are presented to a 32-hour student over 200 epochs.