Stable Gradient-Adjusted Root Mean Square Propagation on Least Squares Problem

Abstract. Root mean square propagation (RMSProp) is a first-order stochastic algorithm used widely in machine learning. In this paper, a stable gradient-adjusted RMSProp (SGA-RMSProp) with a minibatch stochastic gradient is proposed, and its properties are studied on the linear least squares problem (LLSP). R-linear convergence of the algorithm is established on the consistent LLSP. The algorithm is also proved to converge R-linearly to a neighborhood of the minimizer for the inconsistent case, with the region of the neighborhood being controlled by the batch size. Furthermore, numerical experiments are conducted to compare the performances of SGA-RMSProp, stochastic gradient descent (SGD), and the original RMSProp with different batch sizes. The faster initial convergence rate of SGA-RMSProp is observed through numerical experiments, and an adaptive strategy for switching from SGA-RMSProp to SGD is proposed, which combines the benefits of these two algorithms.