Reaaliaikaista kolmiulotteista semanttista segmentointia muunnetun todellisuuden head-up-näytöille

Augmented reality head-up displays are becoming increasingly common in vehicles. State-of-the-art AR HUD systems can render three-dimensional content at varying depths in the driver’s field of view. To place AR visualizations accurately on the HUD without visible lag, the perception pipeline must deliver reliable scene understanding in real-time. Semantic segmentation is a key computer vision task to produce AR HUD visualizations, assigning a class label to each pixel in an image. For 3D AR HUDs, these labels must be combined with depth to obtain a three-dimensional semantic representation of the surrounding scene. This thesis proposes a real-time 3D semantic segmentation pipeline for AR HUDs. Two semantic segmentation models are trained with knowledge distillation using the Cityscapes dataset. A small test set with 20 frames from Helsinki is constructed with finely annotated labels, to assess segmentation accuracy and robustness with data from outside of the training distribution. 3D semantic segmentation is conducted by combining two-dimensional semantic segmentation with a pixel-wise depth estimate using deep learning and a stereo camera. As the HUD is intended for a vehicle, computational efficiency of a semantic segmentation system is instrumental in making the visuals overlap precisely with the changing surroundings. Thus, latency is a key consideration when choosing suitable deep learning methods and designing the software architecture of the pipeline. Both the end-to-end latency and task-specific compute times are recorded and presented, to assess real-time capability and identify bottlenecks in the system. The implemented 3D semantic segmentation pipeline has the end-to-end latency of around 200 ms, showing a segmented point cloud at nine FPS. Depth estimation is identified as the bottleneck in the system, and should be the primary target of improvement in future work. The semantic segmentation models assessed in this work achieve high segmentation accuracy of around 70% mIoU on the Cityscapes validation set, indicating that model training with knowledge distillation was successful. However, on the Helsinki test set, the models achieve up to around 40% mIoU, showing that model robustness cannot be implied from performance in the training distribution.