State-constrained offline deep reinforcement learning

Offline Reinforcement Learning (RL) aims to find optimal decision-making policies from a fixed dataset. RL is usually applied in an online fashion where the decision-making policy is optimised through interactions with an environment. By trial-and-error, the RL agent seeks to find the best actions to take dependent on the current information. Applying online methods to offline datasets results in a distributional shift between the policy that collected the dataset and the current policy being optimised by the RL agent. This distributional shift causes the value of unseen actions (not in the dataset) to be over-estimated, resulting in poor performance. As a result, offline RL methods are designed to stay close to the actions in the dataset, an approach initially labelled as batch-constrained. In this thesis, a new approach to offline RL is introduced. Rather than constricting explicitly to the state-action pairs found in the dataset, the restriction is placed solely on the states themselves. This reduction in restriction aims to find higher-quality policies, by maximising over more seen states than seen actions, while avoiding distributional shift by staying close to in-distribution states. To this end, a data augmentation approach, called Model-based Trajectory Stitching (MBTS), is introduced which alters the sub-optimal trajectories in the dataset by introducing unseen actions between reachable states. As a result of this, the underlying policy of the dataset is shown to improve vastly. Applying MBTS in conjunction with batch-constrained offline RL methods achieves state-of-the-art performance. In addition, State-constrained Q-learning (SCQL) is introduced, which covers the theoretical groundwork for all future practical methods in state-constrained offline RL. It is shown that, under minor assumptions, SCQL should always be preferred to Batch-constrained Q-learning. Two practical deep learning methods of SCQL are then constructed; a one-step and multi-step approach. The multi-step method, attains state-of-the-art performance. Finally, proposals for future work to further develop state-constrained offline RL methods are presented; building on the framework described in this thesis which outlines promising avenues for future research, beyond the current outlook.