Resource Management in Hybrid-Powered HetNets With Two-Timescale Deep Reinforcement Learning

Renewable energy is incorporated to the energy supply for heterogeneous networks (HetNets) to mitigate the high energy consumption and meet carbon-neutral targets. This paper investigates resource management in HetNets with hybrid energy supply, where base stations are powered by renewable energy and power grid, and can share the harvested renewable energy with each other. The objective is to maximize the long-term average utility of the network defined as a weighted sum of system data rate and power grid energy cost. In view of the difference of dynamics characteristics between the radio and energy resources, the problem is formulated by describing energy resources over the large timescale relative to the small one for radio resources. To address the problem without prior knowledge, a two-timescale deep reinforcement learning (DRL)-based optimization framework is proposed, which is structured with two layers of decision-making, including the energy scheduling layer over the large timescale for renewable energy sharing among BSs and power grid energy consumption, and the power control layer over the small timescale for transmit power, respectively. To enhance the performance of the cross-layer collaborative optimization, in the energy scheduling layer an improved soft actor-critic with two-timescale update rule (TTUR-SAC) algorithm is designed with a novel strategy of differentiating the training frequencies of actor and critic networks, and in the power control layer the successive convex approximation method is leveraged to solve the non-convex subproblem for the transmit power. Simulation results demonstrate that under various system configurations our proposed solution outperforms the typical baseline methods.