Real-time semantic segmentation of driving scenes via effective attention-based information fusion and hybrid encoder

Real-time semantic image segmentation approaches require two important properties that should be addressed simultaneously, i.e., extremely accurate segmented outputs and high inference speed. Nonetheless, the semantic segmentation field is dominated by complex deep learning architectures that focus more on the accuracy, and less attention is paid to the real-time processing. In order to attain a low computational complexity such that real-time requirements are considered, we propose a novel lightweight convolutional neural network architecture for the semantic segmentation task. Our model, RTSSNet, uses an encoder especially developed for dense prediction tasks, by incorporating dilated convolutions in the last two stages for better contextual information extraction. Furthermore, we design a lightweight decoder based on attention mechanisms that can effectively recover the spatial details lost during the multiple downsampling operations performed in the encoder. To demonstrate the effectiveness of our method, we conduct extensive experiments on two popular autonomous driving datasets, Cityscapes and CamVid. Compared with other state-of-the-art techniques, our models achieve competitive results. In particular, on a single GPU, RTSSNet-L yields [Formula: see text] mIoU at 121.15 FPS on the Cityscapes validation set, and [Formula: see text] mIoU at 120.68 FPS on the CamVid test set.