Object detection and tracking based on image and point cloud instance segmentation matching for robot

To address the challenge of fusing semantic information from camera images with precise spatial measurements from LiDAR point clouds in robotic environmental perception, a fusion-based detection framework is presented based on image-point-cloud instance-segmentation matching. Two-dimensional object masks are first obtained from camera images by an instance-segmentation network and projected onto the 3D point cloud to extract object-specific clusters. These clusters are processed with Euclidean clustering to generate 3D detection boxes, which are then back-projected to the image plane to complete multimodal fusion detection. On this basis, a tracking-gate mechanism is introduced for multi-object data association and lifecycle management, and Kalman filter is employed for object tracking and motion-state estimation. Experiments demonstrate that the proposed approach effectively integrates image and point-cloud data, achieving accurate and efficient estimation of object position, size, and velocity, and exhibiting robust performance across diverse scenarios.