Out of Control: On the Controllable Augmentation of Driving Scenarios with Bad Visibility Conditions

Accurate detection of vehicles, pedestrians, and obstacles is crucial for autonomous vehicles. Yet, current methods often fail under adverse weather conditions such as fog, rain, or snow, partly due to the limited representation of these scenarios in existing datasets. To address this gap, we present a novel augmentation approach built on the ControlNet framework, which combines multiple conditionings to generate realistic fog, rain, and snow effects on clear-weather images. Our method preserves the scene's overall structure and the form of vehicle objects better than current diffusion-based architectures. By combining multiple conditionings, we achieve realistic fog, rain, and snow augmentations on clear weather images while maintaining scene structure and vehicle objects from the original image. The augmented datasets are used to fine-tune several detection models on real-world fog, rain, and snow scenarios, achieving an Average Precision improvement of up to 5% over baselines that rely on clear-weather training and other AI-based augmentation techniques.