Self-Attention Proximal Policy Optimization for Beam Tracking in mmWave Communications

Millimeter-wave (mmWave) communication systems are highly sensitive to user mobility and environmental changes, often suffering from rapid beam direction shifts and frequent link blockages. These dynamics pose significant challenges to maintaining reliable beam tracking. To enable accurate and robust beam tracking in dynamic environments, we propose a Self-Attention Proximal Policy Optimization (SAPPO) algorithm. The beam tracking task is modeled as a Partially Observable Markov Decision Process (POMDP), where the agent aims to maximize Received Signal Strength (RSS) using only current and previous signal observations, without relying on channel models or complex signal processing. By integrating a multi-head self-attention (MHSA) mechanism with a Gated Recurrent Unit (GRU), the policy network effectively captures temporal dependencies, enhancing the agent’s perception of user motion and channel state variations. To validate the approach, we construct three test scenarios with varying mobility and blockage complexity using the DeepMIMO dataset. Experimental results demonstrate that SAPPO consistently outperforms baseline methods such as DQN and PPO in terms of tracking accuracy, stability, and robustness. Notably, in challenging environments with frequent Line-Of-Sight (LOS)/Non-Line-Of-Sight (NLOS) transitions, SAPPO achieves a beam tracking success rate exceeding 95%, highlighting its strong adaptability and reliable performance under dynamic conditions.