A general learning rate improvement strategy for deep neural networks training

Learning rate is the most important parameter for neural network performance optimization, and its selection and theoretical exploration have always been an active research field. Although the learning rate scheduler has been widely observed to improve the performance of model training, the current works do not consider the characteristics of different stages of training and are mostly incompatible with adaptive optimizers. In this work, we construct a general learning rate scheduler based on the different stages of the training process "warm-up - maintain large learning rate - decay", which can meet the changes of learning rate in different training stages while reducing the dependence on hyperparameters. In addition, we theoretically demonstrate the convergence of the proposed general learning rate scheduler on the adaptive optimizers. Compared with the mainstream learning rate scheduler on the current common data sets, we comprehensively evaluate the advanced and robust performance of the proposed learning rate scheduler through a large number of experiments.