Improving Human-Robot Communication in Noisy Environments with Visual Voice Activity Detection

This paper investigates the integration of Visual Voice Activity Detection (VVAD) into human-robot dialogue systems to enhance communication in noisy environments. Usually, speech recognition systems often falter under acoustic interference, limiting their effectiveness in real-world human-robot interactions. By leveraging visual cues, especially lip movements, VVAD supports more accurate speech detection and turn-taking. In this paper, we present a multimodal dialogue system combining VVAD with speech recognition, dialogue management, and context-aware intention recognition. The system is evaluated through a user study involving 30 participants in a noise-rich, simulated restaurant scenario. Results show a substantial improvement in task completion rates and user satisfaction when VVAD is enabled, alongside with a significant reduction in false activations. These findings underscore the value of visual input in robust, socially aware human-robot interaction and suggest VVAD as a critical component in the design of next-generation interactive robots.