Biased Stochastic Gradient Descent under Local Geometry, with Application to Data-Driven LQR

We study stochastic gradient descent (SGD) with biased stochastic gradient oracles for objectives that are only locally smooth and locally gradient dominated, with constants that may grow with the objective value and are required only on a sublevel set. Such oracles arise in data-driven optimization and control, where the gradient must be estimated from noisy data through nonlinear procedures such as model identification or smoothing and truncation. We show that if the oracle bias decays sufficiently fast, even only in expectation, and the second moment of the oracle remains bounded, then, on an event of prescribed probability, the optimality gap converges to zero almost surely with an asymptotic rate, and in expectation with an explicit bound at every iteration that separates the contributions of the initial condition, the oracle bias, and the gradient noise. We illustrate the framework on the linear quadratic regulator (LQR), whose cost satisfies these properties only on sublevel sets and whose policy gradient depends on the unknown system matrices. We consider an indirect estimator, which identifies the system matrices online, and a direct zeroth-order estimator. We characterize the bias and the second moment of both estimators and show how to choose their parameters so that the convergence conditions hold. On an event of prescribed probability, both data-driven policy gradient methods converge to the optimal policy almost surely and in expectation, with guarantees for all iterations rather than at a single prescribed iteration. We compare the two estimators, and numerical experiments on the LQR illustrate the results.