Safe But Not Sorry: Reducing Over-Conservatism in Safety Critics via Uncertainty-Aware Modulation

Ensuring safe exploration in reinforcement learning is essential for real-world deployment. Existing methods, however, often trade safety for performance by producing overly conservative policies or diffuse cost estimates that weaken policy gradients. We propose the Uncertain Safety Critic (USC), which modulates conservatism using critic uncertainty and refines under-covered regions, reducing safety violations by approx 40% while maintaining competitive or higher rewards and cutting cost-gradient error by approx 83%.