PVF-DectNet++: Adaptive Multi-Modal Fusion With Perspective Voxels for 3D Object Detection

To enhance 3D object detection in autonomous driving, recent work combines LiDAR and camera data. However, prior methods often suffer from inadequate image depth information and fixed-weight fusion strategies, limiting semantic extraction and adaptability. PVF-DectNet++ builds on our prior work by employing a perspective voxel projection technique to align both feature types. It introduces an adaptive image semantic feature extraction approach that interpolates image and point cloud intensity into a dense RGB-I multi-channel representation, facilitating the extraction of global, multi-level image features. Furthermore, during the fusion process, a learnable fusion module is designed to address the challenge of individual channels being unable to adapt to varying appearances, colors, and environmental conditions. Experiments on KITTI, nuScenes, and Waymo comprehensively validate PVF-DectNet++. On KITTI, it achieves detection accuracies of 66.3% for pedestrians, 78.8% for cyclists, and 86.8% for vehicles, yielding a 3.56% mAP improvement over PVF-DectNet. Additional tests show further gains, with mAP and NDS increases of 3.8% and 2.6% on nuScenes, and notable boosts in pedestrian and cyclist AP on Waymo. Compared with existing networks, PVF-DectNet++ consistently delivers superior performance, particularly for pedestrian and cyclist detection across diverse benchmarks. The code and model will be released at https://github.com/CQU-AVL/PVF-DectNet-.