Sensor Data Fusion for Adversarial Patch Attack Mitigation

Adversarial attacks on Artificial Intelligence and Machine Learning algorithms are a well studied phenomena when it comes to human recognition and stop sign detection. However, when it comes to marine autonomous vehicles and port and waterway security surveillance, there is a lack of research, especially on the ability of multi-modal sensor data fusion to mitigate these attacks. To address this research gap, we propose using Electro-Optical (EO) and Infrared (IR) sensor data fusion to mitigate patch and noise attacks applied to the MIT Sea Grant Perception dataset. Adversarial attacks, especially patch attacks, are typically applied in the EO domain and we assess how fusing IR can mitigate this attack for autonomous marine vehicle navigation. We develop an object detection and classification pipeline using You Only Look Once (YOLO) and custom feature-level and decision-level data fusion algorithms. We find that EO and IR data fusion mitigates false negatives (or missed detections) better than the single-modality pipeline, making the fusion model more robust when under attack. However, the fusion model trades off some performance under baseline (non-attack) conditions. Despite this, our work demonstrates the feasibility of data fusion to mitigate adversarial attacks, a topic understudied in the maritime literature.