Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach

Efficient traffic signal control is of critical importance for minimizing traffic congestion and enhancing transportation efficiency. Researchers have turned to Reinforcement Learning (RL) for traffic signal control (TSC) due to the dynamic nature of traffic flow. Despite its potential, the real-world application of RL-based controllers is constrained by low sample efficiency and high computational demands. To address these challenges, we propose DTLight, a simple yet powerful lightweight Decision Transformer (DT)-based offline-to-online TSC method that can learn policy from pre-collected offline datasets while maintaining the capability to refine policy with minimal online interactions. Specifically, we propose three novel adaptive knowledge distillation methods to learn a lightweight offline controller from a well-trained larger teacher model to reduce implementation computation. Additionally, we integrate adapter modules to mitigate the expenses associated with fine-tuning, which makes DTLight practical for online enhancement with minimal computation and only a few fine-tuning steps during real deployment. Extensive experiments have been implemented on different traffic scenarios. The results show that DTLight pre-trained purely on offline datasets can outperform state-of-the-art methods in most scenarios. Additionally, online fine-tuning further improves the performance by up to 40.7% over the best online RL baseline methods. Moreover, we introduce Datasets specifically designed for TSC with offline RL (referred to as DTRL). Our datasets and code are publicly available: https://github.com/xingshuaihuang/dtlight.