SMTrack: End-to-End Trained Spiking Neural Networks for Multi-Object Tracking in RGB Videos

Brain-inspired Spiking Neural Networks (SNNs) leverage a sparse, event-driven computational paradigm and have shown great potential for low-power object tracking. However, most existing SNN-based object tracking rely on event camera data, whereas traditional RGB video remains the dominant input modality in real-world applications. Research on RGB-based SNN multi-object tracking, particularly directly trained deep SNN models, is still in its infancy. To address this, we propose SMTrack, the first directly trained deep SNN framework for end-to-end multi-object tracking on standard RGB data. To handle the challenges caused by scale and density variations among objects, we introduce an Adaptive Scale-aware Normalized Wasserstein Distance Loss (Asa-NWDLoss), which dynamically adjusts the normalization factor based on the average object size within each training batch. For the identity association stage, we integrate the TrackTrack to maintain robust and consistent trajectory tracking. Extensive experiments on BEE24, MOT17, MOT20, and DanceTrack demonstrate that SMTrack achieves comparable performance to mainstream ANN-based approaches with only a few time steps. Codes is available at https://github.com/OpenCodeGithub/SMTrack.