A Spectrum Sharing-enabled Collaborative Dynamic Beam Pattern Strategy based on MA-LSTM-DRL for Multi-satellite Multi-beam STINs

Spectrum sharing-enabled multi-beam management is an effective method to enhance global coverage and spectrum efficiency of satellite-terrestrial integrated network (STIN). However, uneven timevarying traffic, high dynamic network topology, and multi-domain interference pose significant challenges for dynamic beam patterning in multi-LEO STINs. In this paper, we propose a spectrum sharing-enabled cooperative dynamic beam pattern framework where multi-beams of multiple LEO satellites can either fully reuse satellite frequency or share remote terrestrial spectrum. We formulate the cooperative dynamic beam pattern strategy as a joint utility maximization problem of throughput, beam queue load, and delay fairness, achieved through beam assignment, sharing model and power allocation optimization for multi-satellites. To solve the non-convex optimization problem, we provide a collaborative and dynamic beam pattern (CDBP) algorithm based on multi-agent deep reinforcement learning (MADRL). Specifically, CDBP transforms the problem to a decentralized partially observable Markov decision processes (Dec-POMDP) and adopts centralized-training-distributed-execution (CTDE) to address challenge of acquiring global information in multi-satellites. Then, long short-term memory (LSTM) network is introduced into DRL of proximal policy optimization (PPO) model, named MA-LSTM-PPO, which captures temporal dependencies of trajectory and beam queue, guiding to drive optimal beam pattern. Extensive simulation results demonstrate that CDBP algorithm can effectively reduce average queuing delay by 5-15ms compared with existing typical schemes, and adapt to uneven traffic demands in STINs.