Absorbing markov decision processes: geometric properties and sufficiency of finite mixtures of deterministic policies

In this paper we investigate several geometric properties of the set of occupancy measures of an absorbing Markov decision process (MDP). In particular, we analyse the structure of the faces generated by a given occupancy measure, together with their relative algebraic interior. We also determine the affine hulls of these faces and describe the associated parallel linear subspaces. It is shown that these structures can be fully characterised in terms of the parameters that define the underlying Markov decision process. Moreover, we establish that the class of finite mixtures of deterministic stationary policies constitutes a sufficient class of policies for uniformly absorbing MDPs with a measurable state space and multiple criteria. We also provide a sufficient condition, stated in terms of the MDP's parameters and a given occupancy measure, under which that measure can be generated by a finite mixture of deterministic policies.