Attention Accelerates Learning of Proximal Policy Optimization Reinforcement Learning in Distributed and Uncoordinated Cognitive Radio Networks

Implementation of Reinforcement Learning (RL) schemes in cognitive radios (CRs) operating in distributed and uncoordinated settings is particularly challenging because of the inherently non-stationarity of the environment. Consequently, CRs require longer learning time to overcome the non-stationary environment. This paper presents a Proximal Policy Optimization (PPO) RL scheme for CR transmit power control in a distributed and uncoordinated environment that has been enhanced by introducing an attention mechanism in both the actor and the critic components. The purpose of the added attention mechanism is to accelerate the learning of the PPO RL scheme. Through an ablation study that considers learning duration and accuracy in policy convergence to an optimal solution, the experimental results indicate that incorporating attention mechanism in both the actor and critic components in a PPO RL scheme can significantly accelerate policy learning, achieving a maximum relative speedup of 0.38. Moreover, despite the challenges of a distributed and uncoordinated environment, the policy accuracy from the attention driven actor-critic scheme exhibits a monotonic increasing trend as a function of exploration step, rising from 85% at 1,000 exploration steps to 92% at 15,000 exploration steps.