Distributionally robust Markov decision process with uncertain transition probabilities: A multiple-priors approach

To mitigate the effects of parameter uncertainty, many studies consider a robust Markov decision process that helps decision makers hedge against worst-case parameters. An inherent drawback of this approach is the difficulty in incorporating priori probabilistic information of unknown parameters, which can potentially be leveraged to find a robust yet not too conservative policy. In this study, we model the uncertainty in transition probabilities using a multiple-priors model and propose a distributionally robust Markov decision process (DR-MDP). The multiple-priors model constructs a set of priors based on the decision makers’ initial probabilistic beliefs about the transition probabilities. Based on historical observations of state transitions, the set of priors is re-evaluated using a Bayes-factor-based criteria and only the admitted priors are updated measure-by-measure using the Bayes’ Rule to obtain a set of posteriors. The objective is to maximize the expected total discounted reward over an infinite horizon under the worst-case posterior. We investigate the asymptotic convergence of the posterior set and provide a rate of posterior convergence. The result of the rate of posterior convergence is critical as it gives insights on how to determine the size of the posterior set. We further examine the asymptotic convergence and the rate of convergence of the optimal value and optimal policy of the proposed DR-MDP. Moreover, we develop an efficient approximation method to solve the proposed DR-MDP and analytically examine the approximation performance. Numerical experiments on a machine replacement problem under four decision-making scenarios show the benefit of the proposed decision-making framework in providing good out-of-sample performance.