WmLSTM: A Plug-and-Play Window-Level mLSTM-Based Temporal Encoder for Robust Visual Tracking

Temporal context modeling constitutes a fundamental issue for robust visual tracking. However, existing approaches are plagued by a critical granularity trade-off: frame-level template update mechanisms inevitably introduce background redundancy due to global frame information aggregation, while token-level propagation mechanisms undermine inherent local spatial correlations via independent feature transmission. To resolve this challenge, we propose WmLSTM, a plug-and-play Window-level mLSTM-based temporal encoder that reconfigures the temporal modeling paradigm for visual tracking. First, window-centric modeling retains intra-window spatial correlations while adaptively suppressing background clutter. Second, we pioneer the application of mLSTM in visual tracking, exploiting its explicit memory architecture that outperforms implicit sequence modeling alternatives (e.g., Mamba). Third, our plug-and-play design enables seamless integration with state-of-the-art trackers with minor computational overhead. Extensive experiments on seven benchmark datasets validate that WmLSTMTrack achieves an excellent balance among accuracy, speed, and parameter efficiency, attaining state-of-the-art accuracy on five benchmarks, superior real-time speed (GPU: 201fps, CPU: 47fps), and compact model size (8.22 M parameters). Moreover, the WMLSTM module consistently enhances the performance of diverse trackers, e.g., real-time FERMT-256: +2.3 points SR75 on GOT-10k, non-real-time EVPTrack-224: +1.8 points P on LaSOText, with merely 30 training epochs. The source code is available at https://github.com/Xiaochen918/WmLSTM.