CSPER: Curiosity-Driven Safety-Prioritized Experience Replay for Mapless Navigation Using Deep Reinforcement Learning

Due to the sparse extrinsic rewards and uniform sampling of experience replay, reinforcement learning approaches suffer from poor exploration and low sample efficiency when applied in complex navigation environments. Moreover, learning a safe behavior policy is still challenging for continuous control of mobile robots. In this article, we propose a curiosity-driven safety-prioritized experience replay (CSPER) that leverages curiosity and safety-prioritized sampling techniques to improve the exploration and sample efficiency without ignoring safety. The state prediction error is taken as the intrinsic reward to augment the normal external reward to further promote sufficient exploration. This often accompanies potentially unsafe actions. To this end, a safety priority criterion based on a temporal difference error is designed to measure the importance of the experience. The proposed replay technique takes both sample safety and efficiency into consideration and achieves efficient and safe navigation tasks. The safe policy is also integrated into the framework to further improve the policy’s safety and robustness. We combine the proposed method with DDPG, SAC, and TD3 and evaluate it in different complex environments. The experimental results show that our proposed method achieves a better performance compared to baseline algorithms in terms of the final performance, sample efficiency, and generalization. Code can be found athttps://github.com/PHZ-897044002/CSPERNavigation-using-Deep-Reinforcement-Learning.