Multimodal Noncontact-Based Modeling of Enervation

The advancement of autonomous vehicle technology has heightened interest in modeling occupant states and behaviors. This paper models occupant circadian state using four modalities: physiological, thermal, linguistic, and acoustic modalities. Moreover, it advances research by developing a fully non-contact pipeline to derive physiological signals from thermal imagery. Linguistic and acoustic features derived from speech are incorporated with the extracted physiological and thermal features to enhance circadian rhythm models and detect enervation states. The proposed approach leverages a multimodal dataset of 36 subjects captured across multiple channels. Comparative analysis between non-contact and contactbased channels reveals that physiological signals from thermal imagery, combined with linguistic and acoustic data, achieve accuracy comparable to or exceeding those of contact-based methods. This non-contact approach achieves accurate identification of energized versus enervated states, reaching 78.6% accuracy. These findings offer a novel framework for integrating unobtrusive sensing technologies in future automotive designs, facilitating next-generation occupant monitoring.