Enhancing Exploration in Actor-Critic Algorithms:An Approach to Incentivize Plausible Novel States

Actor-critic (AC) algorithms are model-free deep reinforcement learning techniques that have consistently demonstrated effectiveness across various domains. Enhancing exploration (action entropy) and exploitation (expected return) through more efficient sample utilization is pivotal to their success. A key strategy for a learning algorithm is to intelligently navigate the environment’s state space, prioritizing the exploration of rarely visited states over frequently encountered ones. However conventional approaches, rarely quantify a novel state’s utility for policy learning, often leading to inefficient exploration. To address this, we introduce the concept of plausible novelty, which uniquely combines state novelty with an estimation of expected benefit. Unlike existing intrinsic reward methods that rely solely on state visitation frequency or prediction errors, this approach strategically incentivizes exploration of states that are both novel and promising for policy optimization, leading to substantial improvements in AC algorithm sample efficiency and overall training performance. Empirical results demonstrate 19% improvement in training return and 30% reduction in standard deviation, averaged across comparisons of three benchmark algorithm pairs in five different environments.