Reducing Noise via Decomposing Gradient in Deep Mutual Learning

Deep Mutual Learning (DML) is a collaborative training paradigm in which multiple networks with identical architectures learn through mutual interaction without relying on a pretrained teacher model. However, aligning the outputs of differently initialized networks often introduces gradient noise, which can hinder training stability and convergence. Moreover, excessive mutual imitation can lead to over alignment, where networks become overly similar in their predictions. To address these issues, we propose a method that decomposes each model’s gradient, removes small magnitude components regarded as noise, and reconstructs the refined gradient for subsequent parameter updates. By suppressing redundant noise components, the proposed approach enables more stable and efficient mutual learning. Experiments on the CIFAR-100 benchmark demonstrate that our gradient decomposition based method consistently outperforms the baseline DML, validating its potential to improve both the stability and diversity of collaborative learning.