Gated side adapters with memory-efficient fine tuning for RGB-T tracking

Multi-modal tracking fuses different domain features to compensate for each other. Due to the large training complexity of the foundation RGB model, several parameter-efficient fine-tuning (PEFT) methods have been presented for RGB-T tracking. Although these PEFT methods effectively reduce the number of parameters, they still require significant resources. They demand more training memory and longer training times, smilar to full fine-tuning. To solve this problem, we propose gated side adapters that remove the backpropagation through the foundation model. Furthermore, we propose a modality fusion module to adaptively integrate the foundation model with side model to overcome the domain gap. We reduce training time by 29.2% and memory usage by 51.7%, compared to full-fine tuning.