Driver Maneuver Recognition Through Semantic Representation and Spatiotemporal Structural Information

In intelligent transportation systems, consumer electronics like in-vehicle cameras and servers will play a vital role in real-time driver behavior recognition, enhancing both decision-making and system responsiveness. Human-machine shared driving, a core component of future systems, requires precise understanding of driver behavior and intent to support vehicle decisions and real-time traffic control, ensuring networked vehicular safety. The task of driver behavior recognition faces three key challenges: limited data leading to poor model generalization, conventional transfer learning lacking task-specific semantic understanding, and difficulty in capturing fine-grained spatiotemporal behavioral dynamics. To address these, we propose a text-driven driver maneuver recognition approach that combines the semantic analysis capabilities of Large Language Models (LLMs) with the spatiotemporal modeling strength of video Variational Autoencoders (VAEs). Utilizing the CLIP text encoder, our method integrates environmental semantics from both in-car and external settings, capturing specific targets and their movement relationships with greater clarity. Furthermore, Open-SORA’s VAE encoder provides structured video information, enhancing the model’s comprehension of latent video space, and advancing recognition accuracy. Experimental results demonstrate our approach outperforms existing state-of-the-art methods, offering an efficient and robust solution for intelligent transportation. The proposed model supports complex driving scenarios, marking significant progress toward safer, responsive, and intelligent driving systems.