Analysing heavy-tail properties of stochastic gradient descent by means of stochastic recurrence equations

Abstract In recent works on the theory of machine learning, it has been observed that heavy tail properties of stochastic gradient descent (SGD) can be studied in the probabilistic framework of stochastic recursions. In particular, Gürbüzbalaban et al. (2021) considered a setup corresponding to linear regression for which iterations of SGD can be modelled by a multivariate affine stochastic recursion upper X Subscript n Baseline equals upper A Subscript n Baseline upper X Subscript n minus 1 Baseline plus upper B Subscript n X n = A n X n − 1 + B n $X_n=A_nX_{n-1}+B_n$ for independent and identically distributed pairs left parenthesis upper A Subscript n Baseline comma upper B Subscript n Baseline right parenthesis ( A n , B n ) $(A_n,B_n)$ , where upper A Subscript n A n $A_n$ is a random symmetric matrix and upper B Subscript n B n $B_n$ is a random vector. However, their approach is not completely correct and, in the present paper, the problem is put into the right framework by applying the theory of irreducible-proximal matrices.