Voice interaction under cognitive load: Non-verbal auditory inputs for automated vehicles

In automated vehicles (AVs), the driver’s dual role as system supervisor and participant in Non-Driving Related Activities (NDRAs) poses a critical human-computer interaction challenge by competing for limited cognitive resources. While large language models have substantially advanced the semantic understanding of voice interaction, the mechanism aspect of how users initiate, maintain, and terminate commands remain underexplored in terms of usability and safety. To address this, our research systematically investigates Non-Verbal Auditory Inputs (NVAIs) and their integration with speech across a three-part experimental study. It progressed from foundational activation feasibility (Experiment 1, N=20) and single-task performance evaluation (Experiment 2, N=35) to quantifying cognitive interference within a dual-task driving simulator (Experiment 3, N=38). Our findings reveal critical trade-offs across the interaction stages. For activation (Experiment 1), users preferred the reliability of traditional wake-up words, whereas NVAIs received higher hedonic ratings but raised accessibility concerns. For continuous control (Experiment 2 and Experiment 3), a hybrid mechanism using a finger snap to terminate a spoken command significantly enhanced control precision without increasing interference to the primary supervisory task. Conversely, a mechanism mapping control to a sustained vowel extension proved unsuitable due to its high cognitive and physiological costs. In conclusion, these results demonstrate that the choice of input mechanism is critical under cognitive load. NVAIs such as snapping show promise as start-up and stop channels in supervisory AV contexts and motivate adaptive, multimodally redundant in-vehicle systems that better balance efficiency, user comfort, and safety.