DepthSort: Multi-Object Tracking Optimization for Unreliable Detection With Depth Information

Multiple Object Tracking (MOT) often struggles in crowded scenes due to frequent occlusions, where low-confidence detections are typically discarded, leading to track fragmentation and identity switches. To address this, we propose DepthSort, a plug-and-play, training-free framework leveraging estimated depth to enhance robustness under occlusion. Unlike methods relying solely on confidence scores, our approach evaluates the reliability of low-confidence detections using relative depth cues derived from bounding box geometry, enabling the preservation of occluded true objects. We introduce three depth-dependent modules: (i) DDS for screening reliable low-confidence detections, (ii) DA enhances association accuracy by incorporating depth cues into association stage and (iii) DTI enables early initialization of occluded and emerging objects. Extensive experiments show that DepthSort achieves state-of-the-art performance on MOT17 and MOT20 benchmarks, setting new records in HOTA and IDF1 metrics. Without appearance features, it runs at 62.6 FPS on MOT17 and 30.3 FPS on MOT20, enabling real-time deployment. Thanks to its simplicity and modularity, DepthSort exhibits strong generalization across diverse tracking frameworks and challenging environments.