DSSM: Dynamic Streaming Pipeline With Spatiotemporal Model Toward Remote Sensing Object Detection

In the domain of visual perception tasks for Unmanned Aerial Vehicle (UAV) remote sensing, conventional object detection algorithms suffer severe challenges arise from drastic variations in perspective and complex backgrounds, proving particularly problematic for the identification of occluded and fast-motion targets. To address these issues, this paper proposes a novel dynamic streaming architecture that encompasses the following three solution approaches: 1) We have established a revolutionary Dual-Stage Streaming data pipeline (e.g., DSS Dataloader, DSS Sampler, and DSS Transform) to perceive target locations with more enriched spatiotemporal states, leveraging spatial pipeline to expedite rapid convergence on static representations, and applying temporal pipeline to facilitate fine-grained refinement by inject dynamic spatiotemporal details; 2) We have devised a novel selective memory module (SMM) that utilizes decoupled gating units to adaptively encode and store critical spatiotemporal features pertinent to the target from historical frame sequences, while furnishing abundant contextual priors and endows the model with the capacity to capture long-range temporal dependencies; 3) We have proposed a spatiotemporal loss function designed to constrain and guide the streaming model through state smoothing constraint, motion response constraint, and gating conflict penalty, thereby improving the localization performance of the model. Moreover, we introduce a comprehensively revised version of the VisDrone-VID dataset prompted by the identification of numerous instances of disorganized bounding box annotations and erroneous labels within the original compilation. Finally, extensive experiments demonstrate that the Dynamic Streaming Pipeline with Spatiotemporal Model (DSSM) outperforms all other state-of-the-art detection methods, achieving 14.4%, 77.6% and 58.4% mAP@0.5:0.95 on the VisDrone-VID, EMRS-Frame and EMRS-Event dataset. The updated VisDrone-VID dataset and our code are realeased at https://github.com/Jushl/DSSM.