DreamerVFD: Generating Interpretable Plans With Model-Based Reinforcement Learning and Value Function Decomposition

Despite the success of Deep Reinforcement Learning (DRL) in high dimensional control tasks, lack of model transparency and performance guarantees limit their applicability to real-world, safety-critical tasks, such as satellite operations. DRL agents are "black box" models, meaning that their size and complexity render them impossible for humans to interpret. Proper calibration of trust for uninterpretable DRL agents is a major hurdle for safe and successful deployment of DRL agents in satellite operations. Explainable Reinforcement Learning is a growing subfield of Reinforcement Learning (RL) that focuses on generating human-interpretable explanations for the behavior of black box DRL agents. Although methods such as saliency maps, attention visualization, interpretable surrogate models, value function decomposition (VFD), and policy summarization have made progress toward the goal of increasing human-RL alignment, most applications of these techniques focus on subjective, post hoc analysis of a DRL agent's behavior. If these explanations could be generated before the agent carries out its desired action, human operators would have the opportunity to review, validate, and even amend the agent's plan of action. This research introduces DreamerVFD, a novel DRL approach exploring the application of VFD and Model Predictive Control planning to the Model-Based Reinforcement Learning algorithm DreamerV3. The explicit visualization of an agent's plan alongside its value network's decomposed predictions, which carry task-specific semantic meaning, serve as ad hoc, pre-action explanations. This work evaluates the performance, sample efficiency, and generalizability of two variations of planning algorithms applied to a value function decomposed version of DreamerV3 and the accuracy of the interpretable plans these algorithms generate in a set of satellite control tasks. The results from these experiments show that DreamerVFD with Cross-Entropy Method planning outperforms other algorithms in simulated satellite operations by achieving higher or comparable success rates and rewards while reducing collisions by an order of magnitude. DreamerVFD with Cross-Entropy Method planning demonstrated a significant advantage in zero-shot generalization to satellite control tasks with altered reward functions. Further analysis revealed that compounding prediction error during planning varies depending on the environment and limits the accuracy of generated explanations.