Signal Timing and CAV Trajectory Joint Control Under Mixed Vehicular Environments With Hierarchical Proximal Policy Optimisation

ABSTRACT This paper proposes a novel Signal‐Vehicle Cooperative Control framework (SVCC‐HPPO) based on the improved Hierarchical Proximal Policy Optimisation (H‐PPO) algorithm to jointly optimise traffic signal timing and Connected and Autonomous Vehicle (CAV) trajectories under mixed vehicular environments with both CAVs and Human‐Driven Vehicles (HDVs). A hierarchical hybrid action space is designed to effectively constrain CAV acceleration and signal timing adjustments while explicitly accounting for car‐following dynamics near intersections, enabling flexible exploration within physical limits. The hybrid actor‐critic architecture facilitates simultaneous optimisation of discrete and continuous actions through parallel actors guided by a global critic, balancing optimization effectiveness with training stability. A multi‐objective reward function simultaneously minimises vehicle delay and fuel consumption and maximises ride comfort. The core improvement involves a layered entropy regularisation strategy within the H‐PPO algorithm, which separately manages discrete and continuous entropy to enhance exploration efficiency and stability across hybrid action dimensions. Real‐world intersections evaluation results demonstrate that SVCC‐HPPO significantly outperforms benchmark methods TRANSYT and DRL‐based algorithms, achieving reductions of up to 46.3% in delay, 59.5% in queue length, and 52.9% in fuel consumption, alongside a 177.4% improvement in average speed. Performance gains are further enhanced with shorter optimisation intervals and higher CAV penetration rates.