Cracks in the vision system: Vandalism-induced occlusion attacks in autonomous vehicles

Autonomous Vehicles (AVs) rely on machine learning (ML) models for real-time perception and decision-making. However, these models are vulnerable to adversarial attacks, particularly physical-world occlusions, caused by environmental interference or malicious intent; many of which stem from vandalism on AVs due to personal and political agendas. This paper proposes a novel physical adversarial attack called the Vandalism-induced Occlusion Attack (VOA) that can be employed by adversaries to compromise the ML-based perception systems in AVs. We develop multiple VOA attack vectors and evaluate their impact on state-of-the-art object detection models in AVs. Experimental results reveal that structured VOAs such as bottom-top lead to the least degradation in performance, with YOLOv8l maintaining an F1-score of 0.80 and YOLOv5x reaching 0.78 at 30% severity. In contrast, unstructured VOAs like random or targeted attacks cause substantial performance drops-Faster R-CNN's F1-score falls as low as 0.09 under targeted occlusion. These findings underscore the need for VOAaware training and robust feature extraction strategies in AV perception systems.