Reinforcement learning based on-ramp merging decision-making and control strategy for connected and automated vehicles in mixed traffic

In the challenging context of mixed traffic environments, ensuring that connected and automated vehicles (CAVs) can make flexible merging decisions that balance optimality and computational efficiency is crucial. Particularly in the presence of human-driven vehicles (HDVs), their uncertain movements make this task even more intricate. To tackle this issue, our study proposed a CAV merging decision-making and control strategy combining deep reinforcement learning (RL) and analytical optimal control law. First, we formulated the merging problem as an optimization task with objectives encompassing energy efficiency, effectiveness, and safety. The Markov decision process is employed to model the decision-making process for CAV on-ramp merging. Subsequently, the deep Q-network (DQN) was employed to dynamically determine a flexible desired merging position, while the analytical optimal control law, based on Pontryagin's minimum principle (PMP), computed real-time optimal control inputs and corresponding energy consumption based on this desired position. The energy consumption derived from PMP served as a reward for DQN, guiding its decision-making policy updating. Simulation results demonstrated that the proposed strategy can safely and efficiently guide CAVs through merging operations, while showcasing a degree of traffic scenario generalization.