Offline Generalized Actor-Critic with Distance Regularization

In order to address the issue of overly conservative offline reinforcement learning (RL) methods that limit the generalization of policy in the out-of-distribution (OOD) region, this article designs a surrogate target for OOD value function based on dataset distance and proposes a novel generalized Q-learning mechanism with distance regularization (GQDR). In theory, we not only prove the convergence of GQDR, but also ensure that the difference between the Q-value learned by GQDR and its true value is bounded. Furthermore, an offline generalized actor-critic method with distance regularization (OGACDR) is proposed by combining GQDR with actor-critic learning framework. Two implementations of OGACDR, OGACDR-EXP and OGACDR-SQR, are introduced according to exponential (EXP) and open-square (SQR) distance weight functions, and it has been theoretically proved that OGACDR provides a safe policy improvement. Experimental results on Gym-MuJoCo continuous control tasks show that OGACDR can not only alleviate the overestimation and overconservatism of Q-value function, but also outperform conservative offline RL baselines.