MCMC-Driven Learning

MCMC typically involves simulating a Markov chain with a fixed kernel to obtain approximate draws from a fixed target distribution. But in recent years, MCMC has been increasingly used in situations where the target, kernel, or both involve parameters that are tuned using the states generated by the Markov chain. Examples abound: adaptive MCMC, variational inference (both forward and reverse KL), transport-assisted MCMC, surrogate-likelihood inference, coreset MCMC, Markov chain gradient descent, and variational parallel tempering are all of this form. This chapter begins by casting all of these problems in the unifying framework of Markovian optimization-integration (MOI) problems. It then provides practical strategies for solving MOI problems, a basic introduction to the theory of convergence of stochastic algorithms for solving MOI problems, and a detailed example studying the use of MCMC for finding a variational approximation to a target distribution of interest.