Self-distillation language models for neural machine translation in low-resource scenarios

Neural Machine Translation (NMT) encounters parameter and computational constraints in edge deployment. While conventional knowledge distillation improves compact models via large teachers, its pre-training overhead is prohibitive. We propose a self-distillation framework eliminating external teachers by utilizing early training checkpoints (k-th step) as instantaneous teachers for intra-model knowledge transfer. The framework proposed: (i) a dual distillation that combines output-layer alignment using temperature-adjusted soft labels with intermediate-layer representation consistency through MSE (Mean Squared Error) loss (ii) an adaptive composite loss dynamically weighting cross-entropy and KL divergence with epoch-wise temperature adjustment. Experiments on IWSLT low-resource splits (EN-DE/FR) show a 4-layer Transformer (hidden dim 256) achieving +3.2/+5.1 BLEU over non-distilled baselines, nearing 12-layer teacher performance while reducing training time by 37% and GPU memory by 42%. The method achieves end-to-end training with 100k sentence pairs—no pre-training, data augmentation, or re-distillation required—confirming mobile deployment feasibility. This work establishes an efficient paradigm for low-resource NMT compression, bridging the performance compression gap and advancing teacher-free distillation research.