Dynamic Sparse Training Breaks the Correlation between Plasticity and Reward in Deep Reinforcement Learning

Training deep reinforcement learning (DRL) agents remains brittle, a difficulty often attributed to a loss of network plasticity, commonly monitored using proxies such as dormant neuron prevalence and representational rank. Dynamic sparse training (DST), which continually prunes and regrows network connections, can be motivated as a mechanism for preserving plasticity under non-stationary learning problems. This thesis examines whether DST reliably improves standard plasticity proxies in value-based DRL, and whether such improvements align with cumulative reward. In a controlled supervised learning setting with induced non-stationary targets, we con"rm that DST effectively mitigates plasticity loss. Following this we conduct an empirical study comparing dense, static sparse, gradual magnitude pruning (GMP), and DST agents across modern value-based algorithms, including deep Q-network (DQN) and parallelised Q-network (PQN), on the Arcade Learning Environment (ALE). Our results reveal a consistent de-correlation between plasticity and reward. Although DST improves standard plasticity proxies most effectively, these improvements do not result in superior rewards. In contrast, GMP achieves the highest performance by maintaining intermediate plasticity levels, which surpass dense baselines but remain below those of DST. These results support the hypothesis that the mechanism of continual adaptation in DST can induce a level of plasticity that is detrimental to value-based learning. Ablations over sparsity and mask evolution hyperparameters show that topological changes with stronger plasticity proxies do not reliably increase return, the highest return regimes vary across environments, and that both overly conservative and overly aggressive topology updates can impair performance. These findings motivate future work on developing more effectively, environment dependent topological evolution strategies for DST in DRL.