SimAM-KD: Attention-Enhanced Knowledge Distillation for Efficient Image Classification

Deploying deep neural networks on resource-constrained edge devices remains challenging due to their high computational requirements. We propose SimAM-KD, a comprehensive framework that combines attention mechanisms, knowledge distillation, and structured pruning to create efficient yet accurate image classification models. Our approach adopts a parallel attention configuration integrating the Simple Attention Module (SimAM) and Coordinate Attention (CA) with MobileNetV3, and is the first to combine this parallel attention with knowledge distillation for general image classification. Extensive experiments on CIFAR-10 and CIFAR-100 demonstrate that SimAM-KD achieves 88.02% accuracy on CIFAR-10, representing a +4.49% improvement over the vanilla MobileNetV3-Small baseline. Furthermore, our pruning strategy reduces model parameters by 29% while maintaining 87.77% accuracy. Ablation studies reveal that temperature T = 4.0 and distillation weight α = 0.7 yield optimal results. The framework also generalizes to CIFAR-100, achieving +4.69% improvement. Our results establish SimAM-KD as an effective approach for developing lightweight models suitable for edge deployment.