Almost Sure Convergence of Stochastic Approximation: An Interplay of Noise and Step Size

We study the almost sure convergence of the Stochastic Approximation algorithm to the fixed point $x^\star$ of a nonlinear operator under a negative drift condition and a general noise sequence with finite $p$-th moment for some $p > 1$. Classical almost sure convergence results of Stochastic Approximation are mostly analyzed for the square-integrable noise setting, and it is shown that any non-summable but square-summable step size sequence is sufficient to obtain almost sure convergence. However, such a limitation prevents wider algorithmic application. In particular, many applications in Machine Learning and Operations Research admit heavy-tailed noise with infinite variance, rendering such guarantees inapplicable. On the other hand, when a stronger condition on the noise is available, such guarantees on the step size would be too conservative, as practitioners would like to pick a larger step size for a more preferable convergence behavior. To this end, we show that any non-summable but $p$-th power summable step size sequence is sufficient to guarantee almost sure convergence, covering the gap in the literature. For the regime $p \in (1, 2)$, we show that using the $\|x-x^\star\|^p$-like Lyapunov function and applying the Holder-smoothness bound suffice. For $p > 2$, such an approach is no longer applicable, and therefore, we introduce a novel iterate projection technique to control the nonlinear terms produced by high-moment bounds and multiplicative noise. We believe our proof techniques and their implications could be of independent interest and pave the way for finite-time analysis of Stochastic Approximation under a general drift condition. As a starting point, we establish almost sure convergence results for non-expansive operators under a multiplicative noise condition for $p \in (1,2]$ and an additive noise condition for $p \in (2,\infty)$.