Research on multi-modal target recognition and tracking fusion algorithm for pod vision system

In this paper, a multi-modal fusion algorithm is proposed for the target recognition and tracking of pod vision system in complex environment. The algorithm designs a multi-modal feature extraction module by integrating three modal data of visible light, infrared light and lidar. The improved YOLOv7-Tiny, lightweight ConvNeXt and Voxel-FPN networks are used to extract each modal feature respectively, and they are unified into a 256-dimensional feature tensor through channel compression. The dynamic weight fusion module adaptively adjusts the weights of each mode according to the environmental parameters to improve the environmental adaptability. Spatio-temporal attention tracking module combines spatial attention mechanism and LSTM time sequence unit to enhance the stability and continuity of target tracking. The anti-occlusion re-recognition module optimizes the re-recognition ability of the target in the occlusion scene through multi-scale local feature extraction and improved Hungarian algorithm. In addition, this paper adopts model pruning, knowledge distillation and GPU-FPGA collaborative acceleration strategy to control the processing delay within 23.7ms/ frame to meet the real-time requirements of pod platform. The experimental results show that the algorithm performs well on DAVIS-MODIS data set, and the [email protected] of target recognition reaches 89.3%, 83.1%, 81.9% and 85.6% respectively in sunny days, foggy days, nighttime and bright light conditions, the MOTA of multi-target tracking reaches 82.9%, and the ID Switch Rate is as low as 2.1 times/minute, which is significantly better than the existing methods.