Stochastic Optimization of Non-Convex Objectives with Adaptive Learning Rate Annealing

This paper investigates the challenges of optimizing high-dimensional, non-convex objective functions, a common problem in various fields including machine learning and operations research. We explore the efficacy of stochastic gradient descent (SGD) algorithms with adaptive learning rate annealing schedules. Specifically, we propose a novel annealing scheme that dynamically adjusts the learning rate based on the observed gradient variance and curvature information. We provide theoretical justification for the proposed approach and demonstrate its superior performance compared to existing methods on benchmark optimization problems, including those arising from representation learning tasks.