Risk-Aware Multi-Agent Safe Reinforcement Learning for Autonomous Vehicle On-Ramp Merging

Ensuring safety while achieving better efficiency remain significant challenges for autonomous vehicles. Most current research treats safety metrics as components at the same level as efficiency, environmental friendliness, and comfort in the reward function, overlooking the fact that in driving behavior, safety should be a constraint that takes precedence over other components. Additionally, driving risk prediction is rarely considered, resulting in either overly aggressive or excessively conservative policies. We model on-ramp merging in mixed traffic as a multi-agent cooperative framework and propose a reinforcement learning algorithm based on primal-dual optimization. The technique features an upper-level decision system formulated as a partially observable constrained Markov decision process that incorporates risk prediction and lane-changing efficiency, complemented by a lower-level knowledge-enhanced risk prediction system. An action mask module is introduced to filter out unsafe and invalid actions. Simulation experiments demonstrate that this approach significantly reduces unsafe behaviors without compromising traffic efficiency, while the combination of risk prediction and action masking also reduces unsafe behaviors during training, offering a promising solution for autonomous driving sim-to-real applications.