Directed Graphs in Reinforcement Learning: A Benchmark for Balancing Efficiency and Fidelity in Autonomous Vehicle Testing

The effectiveness of existing testing methods is under scrutiny due to significant limitations, notably the discrepancy between action distributions and actual distributions caused by inadequate environmental understanding. Additionally, the lack of a scheme prioritizing fidelity while coordinating testing efficiency results in considerable divergence between test and natural scenarios. To address these issues, this paper proposes a Directed Graph Reinforcement Learning approach with action constraint optimization (DGRL) to generate critical scenarios, balancing efficiency and fidelity. By incorporating directed graph convolutional networks, the model encodes environmental states within the observation interval, providing spatiotemporal insights and reducing action distribution discrepancies. Furthermore, it constructs an unbiased estimation reward considering action constraints by sampling action confidence intervals, filtering out distorted actions, thus balancing efficiency and fidelity. DGRL was trained using the highD dataset, demonstrating robust acceleration performance, with interquartile range ($IQR$) of 3.1 and first quartile ($Q_{1}$) of 75.6 in the acceleration ratio distribution, representing the bandwidth and baseline, respectively. The model also achieved high fidelity, with scenario discrepancies compared to natural scenarios reduced by 85.4% ($Q_{1}$) and 46.5% ($IQR$) relative to GAIL, which considers the rationality of driving behavior. Here,$Q_{1}$represents the baseline of scenario discrepancy, and$IQR$denotes the distribution bandwidth of scenario discrepancy. Additionally, there was 69.7% reduction in the upper limit of the 95% confidence interval, indicating a significant decrease in maximum scenario discrepancy. Deployment on an intelligent connected hardware-in-the-loop testing platform validated DGRL’s effectiveness and applicability in real-world.