Distributed Beam-Hopping Scheduling for LEO Mega-Constellation Networks Based on Hierarchical Multi-Agent Deep Reinforcement Learning

Beam hopping (BH) is crucial for managing communication resources in large-scale multi-satellite, multi-beam Low Earth Orbit (LEO) constellation networks. However, achieving high-dynamic multi-satellite cooperative BH in LEO mega-constellations while satisfying multi-dimensional resource allocation demands remains a significant challenge. To address the interdependence of long-term coverage stability and short-term resource volatility, this paper proposes a distributed Beam Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework. Within this framework, a hierarchical Duplex Dueling Multi-agent Q-learning (QPLEX) architecture decouples the high-dimensional joint optimization problem into two tractable layers: satellite-level cell association and beam-level resource scheduling. The former handles coverage planning under load-balancing and interference constraints, while the latter manages dynamic resource allocation amid spatiotemporal traffic variations. Additionally, Chemical Reaction Optimization (CRO) and Cross-Layer Attention (CLA) mechanisms are introduced to address the discrete beam interference problem and the inter-layer information isolation challenge, respectively. Simulation results show that the proposed method, BH-HMARL, performs better than baseline methods. Further results confirm its scalability, with performance remaining stable despite variations in the number of satellites, beams, and cells. Moreover, BH-HMARL achieves real-time scheduling with execution time under 5ms per agent, satisfying slot duration constraints.