Approximation of Markov chain expectations and the key role of stationary distribution convergence

Abstract Consider a sequence $P_n$ of positive recurrent transition matrices or kernels that approximate a limiting infinite state matrix or kernel $P_{\infty}$ . Such approximations arise naturally when an infinite state Markov chain is truncated and replaced with a finite-state approximation. Another situation of such approximations is when $P_{\infty}$ is a simplified limiting approximation to $P_n$ when n is large. In both settings, it is often verified that the approximation $P_n$ has the characteristic that its stationary distribution $\pi_n$ converges to the stationary distribution $\pi_{\infty}$ associated with the limit. We show that when the state space is countably infinite, this stationary distribution convergence implies that $P_n^m$ can be approximated uniformly in m by $P_{\infty}^m$ when n is large. We show that this ability to approximate the marginal distributions at all time scales m fails in continuous state space, but is valid when the convergence is in total variation or when we have weak convergence and the kernels are suitably Lipschitz. When the state space is discrete (as in the truncation setting), we further show that stationary distribution convergence also implies that all the expectations that are computable via first transition analysis (e.g. mean hitting times, expected infinite horizon discounted rewards) converge to those associated with the limit $P_{\infty}$ . Simply put, we show that once we have established stationary distribution convergence, we can immediately infer convergence for a large range of other expectations.