Markov Decision Processes: On the Convergence of the Monte-Carlo First-Visit Algorithm
Abstract
Abstract. We consider the Monte-Carlo first-visit algorithm, of which the goal is to find the optimal control in a Markov decision process with finite state space and finite number of possible actions. We show its convergence when the discount factor is smaller than [Formula: see text].
Fields of Study & Subjects
Optimization and Search Problems
Game Theory and Applications
Markov Chains and Monte Carlo Methods
Publication details
- Journal / source
- SIAM Journal on Control and Optimization
- Volume
- 64
- Issue
- 3
- Pages
- 1672–1697
- Publication date
- 3 June 2026
- Language
- EN