Advancing autonomous driving systems: A 3-dimensional U-Net framework for object detection via fusion of camera and LiDAR sensors

Object recognition is essential for autonomous cars, and the amalgamation of camera and light detection and ranging (LiDAR) sensor data has emerged as a pivotal method for accurate three-dimensional (3D) object recognition. Contemporary algorithms face challenges with fragmented data, high processing costs, insufficient resolution, and limited dynamic information. This study presents a novel approach utilising 3D U-Net deep learning for precise 3D object detection and localisation by integrating camera and LiDAR data. The process involves obtaining and preprocessing camera and LiDAR data, utilising a geometric 3D frustum method to extract 3D information from LiDAR based on 2D camera bounding boxes, and training a You Only Look Once version 4 (YOLO v4) network to recognise these boundaries in camera images. The detected images are combined with LiDAR data, and a deep U-Net network is utilised to define 3D bounding boxes. Performance is assessed at various noise levels (0 %, 1 %, 2 %, 5 %, and 10 %) in the composite images. This method leverages the benefits of both sensors to better object recognition across diverse shapes and sizes, even in challenging situations, signifying a significant progression towards safer and more reliable autonomous vehicles with improved situational awareness in intricate urban environments.