Reinforcement Learning for Driving Policy Generalization via Risk-Aware Expert Policy and Conditional Diffusion

In real-world autonomous driving applications, reinforcement learning (RL) faces several persistent challenges including limited critical training data, low exploration efficiency, and poor policy generalization. These limitations often result in slow convergence or stagnation in local optima, undermining the safety and generalization of intelligent vehicles. In order to address these issues, we propose a generation-enhanced RL framework for challenging decision-making scenarios, which comprises three stages: data acquisition, data utilization, and data generation. Firstly, a risk-aware expert policy is developed to guide exploration on regions proximal to the boundary between safe and unsafe actions during the early phases of training. Secondly, we propose a hierarchical dynamic prioritized replay mechanism to enhance the utilization of collected experiences. The replay buffer dynamically ranks experience samples based on their relevance to current policy updates. By weighting essential transitions more heavily during replay, the agent gains enhanced capability to learn from critical scenarios. Thirdly, to mitigate the scarcity of high-risk transitions in the training data, we propose a conditional generation method based on diffusion model. This model synthesizes diverse and structurally relevant transitions with statistical realism, which is concentrated near the policy decision boundary. By employing the high-risk metric as conditional input, the generation model supplements the replay buffer by covering distributional gaps, leading to policy generalization in rare but safety-critical scenarios. Experimental results demonstrate that our method significantly accelerates policy learning and improves driving performance under complex scenarios. The learned policies also outperform existing baselines, indicating their advantages in terms of safety and generalization.