RaCFusion: Improving Camera-Based 3D Object Detection via Radar-Assisted Hierarchical Refinement

Cameras and radar sensors are complementary in 3D object detection in that cameras specialize in capturing an object's visual information while radar provides spatial information and velocity hints. Existing radar-camera fusion methods often employ a symmetrical architecture that processes inputs from cameras and radar indiscriminately, hindering the full leverage of each modality's distinct advantages. To this end, we propose RaCFusion, a radar-camera fusion framework that leverages the camera stream as the main detector and improves it via Radarassisted hierarchical refinement. Technically, the Radar-assisted refinement is performed via two specifically designed modules. Firstly, in the Radar-assisted Query Generation module, the initial object queries of the image branch are augmented with the spatial information obtained from radar data, formulating enhanced hybrid object queries. These hybrid object queries are used to interact with the image features in the transformer decoder to generate object-centric query features. Subsequently, within the Radar-assisted Velocity Aggregation module, these query features undergo further refinement through the incorporation of Radar-assisted velocity features. These velocity features are meticulously learned from the nuanced relationships between the queries and radar features, thereby diminishing the error in velocity estimation by utilizing the valuable velocity clues from the radar sensor. RaCFusion achieves competitive performance among radar-camera-fusion 3D detectors on the nuScenes benchmark. The project is athttps://jessiew0806.github.io/RaCFusion/.