Research on multi-modal driver behavior recognition method based on heterogeneous data learning

Driver behavior recognition refers to the technology of determining the real-time status of drivers through video data or other sensor data. Multimodal driver behavior recognition refers to the technology of determining the real-time status of drivers through heterogeneous data such as video and audio. This article focuses on the research of multimodal driver behavior recognition. A novel deep learning network based on Cross Convolutional Neural Network is proposed for information exchange between video and audio data, effectively reducing the structural differences of heterogeneous data and improving model generalization and robustness. At the same time, a Cross Complementary Learning network is proposed for multimodal driver behavior recognition algorithms. This algorithm simultaneously delivers video and audio representations, trains the collaborative relationship between video images and audio data, and overcomes the differences within and between modalities.