Evaluating the adversarial robustness of detection transformers

Abstract Robust object detection is critical for autonomous driving and mobile robotics, where accurate recognition of vehicles, pedestrians, and obstacles is essential for ensuring safety. Despite the advancements in detection transformers (DETRs), their robustness against adversarial perturbations remains underexplored. This paper presents a comprehensive evaluation of the DETR model and its variants under both white-box and black-box adversarial attack settings, using the MS-COCO and KITTI datasets to cover general-purpose and autonomous driving scenarios. We adapt a suite of popular white-box attacks in image classifications including FGSM, PGD, C&W and AutoPGD to object detection and assess DETR’s vulnerability under these threats. In addition, we introduce a Modified C&W attack tailored to the DETR architecture that exploits intermediate decoder losses to induce misclassification with minimal perturbations. Our results demonstrate that DETR models are significantly susceptible to the entire range of attacks, and that intra-network transferability between DETR variants is high, whereas cross-network transfer to Faster R-CNN is more limited. We further validate the resilience of Modified C&W against several input purification techniques and visualize self-attention maps to illustrate how adversarial attacks affect the models’ internal representations. These findings reveal critical vulnerabilities in detection transformers under standard and adaptive attacks, underscoring the need for further research to improve the robustness of transformer-based object detectors in safety-critical applications. Codes are available at https://github.com/amirhnazerii/Transformer_ObjDet_Robustness/