Cooperative Content Caching in Vehicular Edge Computing Networks: A Two-Stage Deep Reinforcement Learning Approach

In vehicular edge computing (VEC) networks, by implementing content caching and V2X connectivity, road side unit (RSU) and nearby vehicles can serve as platforms for rapid data retrieval to address mobile traffic explosion. However, due to the dynamic and multi-constrained environment consisting of heterogeneous vehicles and RSU, it is challenging to meticulously plan cooperative caching policies. Additionally, due to the diversity of contents and the mobility of vehicles, the caching policy space is massive, which can be fatal for vehicles with limited computing and energy. In this paper, we formulate cooperative content caching in VEC networks as Markov decision process (MDP), configuring caching policies for vehicles and RSU. Our aim is to minimize Lyapunov drift and long-term delay. To address the massive caching policies, we propose a two-stage deep reinforcement learning (TS-DRL) algorithm. In the first stage, an improved ant colony algorithm is used to generate unilateral suggestions and construct action space to avoid the curse of dimensionality. In the second stage, we combine the Noisy Net and Double Deep Q-Learning Network to avoid overestimating value and efficient exploration problem. Simulation results show that TS-DRL outperforms advanced algorithms in terms of delay and cache hit rate.