Robust Audio-Visual Speech Recognition in Noisy Clinical Environments

Visual acuity (VA) testing is a foundational clinical procedure for assessing vision clarity. Traditionally, VA testing requires patients to read optotypes aloud from a distance, with an assistant present to record the responses. Speech-enabled automation has been explored, however these solutions are highly susceptible to performance degradation from ambient noise and speech crosstalk: a common challenge in real-world, multilane clinical environments. In this work, we introduce a novel multimodal pipeline system that combines audio and visual speech recognition to enable robust and accurate VA transcription even under noisy conditions. By integrating synchronized microphone and camera inputs, our architecture employs advanced noise and crosstalk detection, and utilizes a dynamic decision module to intelligently select between audio-based ASR and lip-reading outputs based on real-time environmental analysis. To further improve accuracy under crosstalk conditions, we introduce a lip-guided audio masking mechanism that suppresses non-target speech by detecting when the patient is actively speaking. This multimodal pipeline represents a significant advancement for automated vision screening, offering enhanced transcription accuracy and reliability where single modality systems fail. Preliminary results demonstrate that the addition of lip-reading achieves a 33% and 16% relative reduction in word error rate (WER) compared to the audio-only baseline on the three-speaker and two-speaker crosstalk visual acuity test sets, respectively.