Sparsity-Based Safety Conservatism for Constrained Offline Reinforcement Learning

Offline reinforcement learning (RL) aims to learn a policy from static datasets, which poses a significant challenge as the data often lacks broad coverage and density. This difficulty is even compounded in safety-critical settings, which additionally impose constraints on the policy. This motivates pessimism, a principle that requires the agent to be conservative and confine the behavior of the policy to actions and states that are well-supported by the data. While pessimism is a foundational principle in offline RL, identifying the appropriate form and magnitude of such conservatism for safety constraints remains an open challenge. Prior works typically resort to bi-level optimization to pessimistically overestimate safety violations, ensuring the updated policy is safe in the worst-case scenario. However, we observe that even worst-case scenario estimations are highly sensitive to hyperparameters; the algorithm tends to under- or overestimate safety violations because it cannot reliably infer information beyond the available data. Furthermore, this approach can introduce significant instability or become computationally intractable in high-dimensional tasks, necessitating a simpler method for inducing conservatism. In this paper, we present a data sparsity-based conservatism metric that selectively overestimates safety constraints in data-sparse regions, while remaining less conservative in data-dense regions. This approach is an intuitive application of the statistical principle that accuracy is highly probable in data-dense regions, whereas uncertainty increases in sparse regions. We demonstrate that this method achieves superior performance and greater simplicity compared to traditional bi-level cost-maximization approaches.