Adaptive Depth Transformer-DQN: Dynamic Computation for Value-Based Reinforcement Learning

Transformer-based Q-networks have demonstrated strong performance in partially observable and memory-intensive reinforcement learning environments. However, existing approaches rely on a fixed computational depth, applying the same amount of computation to all states regardless of their complexity. This leads to unnecessary computation for simple decisions and insufficient reasoning capacity for ambiguous states. We propose Adaptive Depth Transformer-DQN (ADT-DQN), a novel value-based reinforcement learning algorithm that dynamically adjusts Transformer computation depth on a per-state basis. ADT-DQN introduces intermediate Q-value heads and an uncertainty-aware halting mechanism that enables early exits while preserving Bellman consistency. Empirical evaluations demonstrate that ADT-DQN achieves performance comparable to fixed-depth Transformer-DQN while significantly reducing average inference depth, highlighting adaptive computation as a promising direction for efficient reinforcement learning.