DFPR-BEV: Cross-Modal Knowledge Distillation of Decouple Features and Projection Regions for Multicamera 3-D Object Detection

Multi-camera 3D object detection receives extensive attention in autonomous driving because of its advantage in efficiency and cost-effectiveness. However, a performance gap still exists between camera-based detector and LiDAR-based detector. Because camera-based detector cannot extract precise depth and geometric information from the input image data, resulting in obvious projection deviation when transforming features from the perspective view to the bird’s eye view (BEV). We propose a cross-modal knowledge distillation framework (DFPR-BEV) to address this problem, making the camera-based student detector to improve its feature representation by imitating the trained LiDAR-based teacher detector during training. To prevent semantic information from being harmed by directly aligning features during distillation by modal variation, we improve the student model to decouple point clouds feature distillation and image projection position correction. A heatmap-based projection correction method is proposed to correct image feature projection deviation and eliminate conflicting features. A Gaussian mixture region-based method is introduced for comprehensive geometry and depth transmission of point clouds features. Finally, the detection performance is improved by effectively fusing the two distilled features. Extensive experiments on nuScenes benchmark illustrate that DFPR-BEV can significantly improve the performance of the student model, outperforming current distillation methods and other camera-based detectors with equal or lower computational budget. Notably, on the nuScenes val set after distillation, BEVDet’s NDS and mAP are improved by 4.9% and 4.2%, respectively, BEVDet4D’s inference speed is 5× faster than BEVFormer-S with the same mAP and higher NDS.