A relaxation approach to layerwise determination of learning rates in deep neural networks

The determination of learning rate is critical in training neural networks. The optimal one can be determined by the line-search minimization. However, the line search involves matrix-vector multiplications that are expensive in the computational cost and its direct implementation is prohibitive.We introduce two relaxations of the line-search minimization. The relaxations are free from matrix-vector multiplications and can be directly solved. One of them, referred to as naive relaxation, is based on the global Lipschitz constant, a measure of the maximum ratio of the perturbation of gradient relative to a perturbation of weight in forward and backward passes. The other relaxation is based on a local Lipschitz constant, more efficient and adaptive than the former.Conventionally, the weights are concurrently updated with the same learning rate. The coincident updates make the propagations of perturbation collide and become too complicated to be analyzed. Instead, we take a layerwise approach : the weight of one layer is alternatively updated while the others are held fixed. The layerwise approach prevents the collisionand enables us to introduce rigorous convergence results. Empirical result furthermore shows that the optimal learning rates are quite different layerwisely and supports our layerwise approach.