Online Targetless Camera–Lidar Extrinsic Calibration Using Object-Oriented Semantic Feature

Accurate extrinsic parameters are critical for fusing heterogeneous data from cameras and Lidars. To solve the problems of incomplete feature extraction, poor scene adaptability and unstable parameter convergence in conventional automatic calibration methods, this paper proposed an online extrinsic calibration method using object-oriented semantic features, which consisted of a coarse-calibration based on trajectory-consistency and a fine-calibration based on feature-consistency. With the initial parameters from the coarse-calibration, fine-calibration pictorialized the sparse point cloud to obtain its depth, intensity and class attributes simultaneously. Then, the SAM network was used to process both the visual image and the point cloud pseudo-image to extract consistent object-oriented features. To enhance the stability and adaptability of extrinsic calibration to various scenes, an observability-based IoU was proposed to perform the camera-Lidar feature matching. In addition, we designed an adaptability function to evaluate the quality of scene features, thereby ensuring the effectiveness of extrinsic calibration. Results of testing on KITTI dataset and real-world dataset demonstrate that the proposed method can effectively obtain accurate extrinsic parameters in multiple scenes. By comparing with the ground truth, errors of the rotation and translation are less than (0.15deg, 0.05 m), and the variances are less than (2.0×10-3deg2, 3.0×10-4m2), which indicates good potential for engineering applications.