Energy-Efficient Data Offloading for Ultra-Dense Heterogeneous Vehicular Networks: A Multi-Population Mean-Field Reinforcement Learning Approach

For ultra-dense vehicular networks, dynamic resource optimization among a large number of heterogeneous agents is rather challenging. This paper proposes a learning-based resource allocation scheme in a vehicle-assisted mobile edge computing (MEC) network, where the uncrewed aerial vehicles (UAVs) and uncrewed ground vehicles (UGVs) are equipped with the MEC servers to provide the computational offloading services to the ground users with time-varying computing demands. Each self-interested vehicle jointly optimizes its trajectory planning and data offloading policies to maximize the expectation of its locally cumulative energy efficiency under the collision and energy constraints. We model the non-cooperative interactions among the massive co-channel vehicles as a multi-population mean-field game (MPMFG), where each vehicle constructs two types of mean-field terms to model the UAVs’ and UGVs’ population distributions, respectively. We propose a multi-population mean-field parameterized deep Q network (MPMF-PDQN) algorithm to solve the equilibrium among the vehicular servers in a discrete-continuous hybrid action space. The simulation results demonstrate that the proposed algorithm significantly enhances the average energy efficiency of the vehicles compared with the baseline algorithms.