Rollout–Guided Switching to Failure–Trained Policies in Residual Reinforcement Learning

Reinforcement learning (RL) agents achieve strong results in many control domains but may still encounter difficulties in certain challenging scenarios. Residual learning can address this by augmenting the base agent with a secondary network that learns corrective residuals. The training of the second policy can be focused on difficult scenarios via introducing a failure buffer that prioritizes samples where the base agent tends to fail. As a result, the residual policy is conditioned to complete episodes that the base agent alone cannot. To further improve efficiency, our paper introduces a rollout-guided policy switching framework, where the base agent’s future trajectory is projected and the underlying policy is changed to the failure-trained network when an unsuccessful outcome is predicted. We evaluate the proposed method on various environments from the robotic and vehicle control domains. The results show that our approach increases the success rate and overall performance of agents, while also maintaining the control quality.