TSC-DETR: DETR With Transformer-Semantics to CNN Guidance for Small Object Detection

Object detection models based on the Detection Transformer (DETR) architecture suffer from a performance bottleneck due to its attention mechanism’s bias toward high-level semantic information, resulting in insufficient low-level detail feature extraction. Particularly in small object detection, existing DETR models suffer from significant deficiencies in detail representation due to the limited number of pixels in the target region. To address this issue, we propose the Transformer-Semantics to CNN Guidance DEtection TRansformer (TSC-DETR), which leverages high-level semantic guidance and a CNN to enhance low-level edge feature extraction for small object detection. First, we introduce a method to enhance the encoder’s dynamic multi-scale perception capabilities to compensate for the linear attention mechanism’s limitations in describing edge detail. Secondly, we utilize a high-level semantic prior prediction guidance map generated by the Transformer encoder to guide the convolutional network to focus more on positive sample regions. Also, to address the problem of small objects being missed due to the ”flooding effect” of high-level semantics in negative sample regions, we design a prediction guidance map specifically for negative sample regions to improve the model’s ability to recognize small objects. Furthermore, we constructed a multi-scale, multi-granularity detail perception module. This module relies on high-level semantics to focus on positive and negative regions, fusing multi-level information to enhance the expressive power of low-level features in depicting the boundaries and textures of small objects. TSC-DETR outperforms previous CNN-based and DETR-based methods, achieving an AP of 31.5% on AI-TOD and 36.7% on Visdrone2019.The project page:https://github.com/AuroraLove/tsc-detr.