The Social and Temporal Influences on Attention in Audiovisual Scenes

It is extremely rare that we receive any sensory information in isolation. This is particularly true in everyday verbal communication where sound is paired with visual stimuli as we look at our interlocutor. Dialogue is inherently social and collaborative (Garrod & Pickering, 2009). However, it would be exhausting to attend to every input we receive from the environment. The auditory cocktail party describes an effect where listeners selectively attend to a target noise in the presence of competing background noise. This is where selective attention becomes essential. Posner and Petersen (1990, 2012) suggest that there are three main neural mechanisms behind selective attention. The first is alerting, describing arousal towards a potential target stimulus. This can be either short-term and target-driven (phasic) or long-term and related to maintained vigilance (tonic). Secondly, orienting describes when a stimulus is ‘selected’ in the environment and attention is shifted towards it. Last is executive control which describes networks that support focal attention. This is described as the ‘entry to conscious state’. It filters information and resolves conflicting information. The present study aims to use electroencephalography (EEG) and eye-tracking to examine activity related to selective attention to audiovisual (AV) stimuli. The N2pc and N2ac are event-related potentials (ERPs) thought to be markers of selective attention to a target. They are calculated from neural data as the contralateral-to-target minus ipsilateral-to-target difference, making them lateralised. The N2pc emerges from visual stimuli, while the N2ac emerges from auditory stimuli (Gamble & Luck, 2011; Luck, 2011; Luck & Hillyard, 1994). The N2pc is well established, usually appearing 200-300s after stimulus presentation and negatively peaking around 1-2 µV. It is typically maximal over posterior areas near the visual cortex (Luck & Hillyard, 1994; Luck and Kappenman, 2011). There is evidence that the N2ac emerges at a similar time to the N2pc, around 200-300ms (Gamble & Luck, 2011). This too is more negative when the target stimuli are contralateral to an electrode, peaking around 0.5-1 µV. However, its localisation is in anterior brain regions. These two markers of selective attention allow us to neatly compare how auditory and visual information are separately modulated by AV stimuli. Auditory information is processed more quickly than visual information, so it is important to establish its specific role on selective attention in an audiovisual context (Shelton & Kumar, 2010). Klatt et al. (2023) began to explore this area by pairing auditory stimuli with either a blank screen, incongruent, or congruent visual stimuli. An N2ac emerged in each condition, while an N2pc did not emerge in the blank-screen condition. Interestingly, the N2pc emerged but did not significantly differ between when the faces’ lip movements matched the auditory information versus when they did not (congruent and incongruent respectively). This provides further evidence that these ERP components are indicative of auditory and visual selective attention, even in more naturalistic settings. However, the extent to which these ERPs of selective attention are modulated by different features of visual information is not yet established. Klatt et al. (2023) used a blank screen to create an auditory only condition. The present study plans to explore the specific benefit of visual information on selective attention further by manipulating the social and temporal aspects of the visual stimuli. The non-social, non-temporal stimulus will be a scrambled-face condition where square clusters of each face have been randomly moved. The social, non-temporal stimulus will be a passive-face condition featuring a face in motion but without lip-movements (e.g., blinking). The final two conditions are both social and temporal: audiovisual incongruent (AVI) where lip movements are incongruent (‘ba’) to the auditory stimuli (single syllable numbers); and the audiovisual congruent (AVC) where the lip movements are congruent to the auditory stimuli. We hypothesise that neural correlates of visual selective attention (N2pc amplitudes) will be stronger in the presence of social and/or temporal information, versus without and that neural correlates of auditory selective attention (N2ac) will be present in all conditions. We will also conduct exploratory analyses to explore the difference of N2ac amplitude across conditions, and different predictors of amplitudes in the audiovisual congruent condition. By doing this, we hope to provide further evidence that these ERPs are analogous, both representing selective attention in their respective modalities. We also hope to uncover what it is about faces that can provide a benefit to selective attention in listening scenarios.