Safety Guarantees for Neural Network-Encoded Control Policies Through Constrained Reinforcement Learning

Reinforcement learning lacks the capability of yielding control policies with safety guarantees. This constitutes a major hurdle in its application to real-world problems despite these data-driven control algorithms show remarkable success in research settings. Therefore, this paper considers safety-constrained reinforcement learning to reliably provide control policies with guaranteed safety after and especially during the training. First, we derive a set of constraints in the parameter space of the control policy based on explicit support samples in the state space. Then, we introduce a novel property that extends the commonly known concept of control invariance. This enables the construction of safety guarantees for neural networkencoded control policies. We propose to extend the policy optimization problem by these safety constraints to provably enforce system constraint satisfaction for both every intermediate learning step and the final control policy. To solve the nonlinear constrained optimization problem, we utilize a primal interior-point method which facilitates the application of first-order solvers. Finally, we conduct numerical experiments which successfully validate the safety-constrained reinforcement learning control concept. The studies also reveal that the introduction of policy constraints can improve the convergence behavior and reduce the stochasticity of the training process.