On Analysis of Clipped Critic Loss in Proximal Policy Optimization

ABSTRACT Proximal policy optimization (PPO) is a widely used reinforcement learning algorithm valued for its robustness and sample efficiency. Its success is often attributed to the actor's clipped loss, which keeps policy updates within a trust region. In contrast, the critic's clipped loss has received relatively little attention, leaving its consistency with the trust‐region principle unclear. To bridge this gap, we analyze the critic's clipped loss, show its misalignment, and propose a refined loss that enforces trust‐region compliance by construction. Experiments on continuous‐control tasks confirm that the proposed method improves adherence to the trust region.