A Two-Timescale Learning-Based Stochastic Game for Energy-Efficient UAV Trajectory Planning and Replacement in Hierarchical Aerial Computing

The integration of unmanned aerial vehicles (UAVs) and high-altitude platform (HAP) in hierarchical aerial computing presents a promising paradigm for delivering supplementary computational services to mobile IoT devices. Despite the great potential, this system faces the challenge of timescale. Specifically, the dynamic nature of IoT devices and their corresponding task generations may initialize the frequent adjustment of UAVs-HAP task delegation for fullfiling their needs in small-timescale, but UAVs' trajectory planning and replacement can only operate in large-timescale because of the severe overhead in high-speed maneuvers, motivating the multi-timescale asynchronous decision making which may further complicate the system. To this end, we propose a two-timescale optimization framework, aiming to maximize the system energy efficiency of the hierarchical aerial computing. We formulate the problem as two coupled multi-agent stochastic games, capturing UAV-HAP interactions and system uncertainties, and prove the existence of a Nash Equilibrium (NE). A reinforcement learning-based algorithm, UAV trajectory planning, replacement, task delegation and resource allocation (UTRTD), is developed to achieve the NE. Simulation results demonstrate the effectiveness and superiority of the proposed UTRTD over counterparts.