Neural encoding of visual and acoustic speech features during audiovisual narrative listening

The presence of visual cues (e.g. lip movements) can enhance speech perception and the neural tracking of acoustic speech features. It is unclear, however, whether this enhancement relates to measures of behavioural audiovisual benefit. Here, we investigated the neural encoding of acoustically and visually derived speech features, capturing amplitude, temporal, and spectral modulations in the speech signal during narrative listening. We collected MEG and eye-tracking data (n=35) alongside objective (3AFC comprehension) and subjective (intelligibility ratings) measures of story comprehension while participants watched videos of a speaker narrating a story (Varano et al., 2023) in auditory-only (AO), visual-only (VO), and audiovisual (AV). To measure changes in speech tracking due to changes in modality rather than those induced by changes in intelligibility, AO and AV conditions were matched in relative intelligibility using noise-vocoding. We also collected behavioural measures of lipreading ability and audiovisual benefit, speech reception thresholds, and verbal IQ. We found evidence for intelligibility-matched audiovisual enhancement of neural tracking only for visually-derived and spectral features, most prominently over occipital sensors, in line with a perceptual restoration of degraded spectral dynamics (Plass et al., 2020). Next, we used partial correlations to isolate the unique contribution of visual and acoustic speech features to neural tracking, and examined how these relate to individual differences in behaviour. Interestingly, we found that the neural tracking of partial visual features was greatest in individuals who showed less benefit from visual speech behaviourally, possibly reflecting a failure to fuse visual and spectral cues. Additionally, replicating previous work (Aller et al., 2022; Bröhl et al., 2022) we found that the tracking of partial acoustic features in left temporal sensors during silent lipreading was positively associated with increased lipreading ability. Finally, we report preliminary analyses which suggest that these patterns of neural encoding are shaped differentially by mouth-looking behaviour in interaction with behavioural proficiency. Overall, these results suggest that the encoding of visual speech is linked to listening outcomes in a way that depends on condition (AV v. VO) and face-viewing strategy.