Hybrid Deep Neural Network-Based Modeling of Multimodal Emotion Recognition for Novice Drivers

Driver emotion recognition is a crucial method for reducing traffic accidents. Most existing research focuses on experienced drivers as the primary research subjects, overlooking novice drivers, who are inexperienced in driving. However, novice drivers can easily lose control of their emotions due to the high mental load during driving, which can lead to serious traffic accidents. Therefore, to recognize the emotions of novice drivers for timely warnings, we propose an emotion recognition model based on multimodal information. The model consists of a facial feature extraction module, an eye movement feature extraction module and a classifier. The facial feature extraction module uses the ViT-B/16 to extract the facial features of novice drivers. The eye movement feature extraction module is a hybrid network containing Bi-LSTM and Transformer. It extracts eye movement features of novice drivers. Facial features and eye movement features are fused and fed to the classifier. The classifier can output the five major emotion categories of surprise, anger, calm, happy, and other for novice drivers. The experimental results demonstrate that our model accurately recognizes the emotions of novice drivers with an accuracy of 98.72%, surpassing that of other models.