Realism and Neural Dynamics in Face Processing: Evaluating Electrophysiological Markers of Cognitive Mechanisms Across Immersive Virtual Reality and Conventional Laboratory Conditions

Human faces occupy a uniquely central role within the social environment, exceeding all other visual stimuli in their interpersonal relevance. This prominence is mirrored in the extensive body of psychological and neuroscientific research on face processing. However, most existing research relies on traditional laboratory paradigms that present highly standardized, two-dimensional facial images on computer monitors – stimuli that ultimately function only as simplified proxies for the faces encountered in real-world interactions. The limited perceptual realism as well as implausible and improbable appearance of face stimuli represent critical methodological shortcomings of conventional laboratory research. This dissertation seeks to address these limitations through the use of immersive virtual reality (VR). The four empirical studies in this dissertation systematically target specific methodological limitations and investigate how the cognitive mechanisms of face processing reflected in relevant electrophysiological correlates adapt when these constraints are alleviated. Study 1 addresses the central methodological limitation of reduced perceptual realism in laboratory-based face stimuli. To this end, it contrasts the perceptual and conceptual processing mechanisms, reflected in characteristic electrophysiological markers, that are elicited by conventional 2D laboratory faces and realistic virtual faces, and their respective perceptual control stimuli. Studies 2 and 3 extend this focus by targeting the related issue of unrealistic object control stimuli. These studies therefore examine how perceptual, cognitive, and attentional processing of faces versus cars differs between conventional laboratory conditions and realistic virtual environments. Study 4, the centerpiece of this dissertation, integrates perceptual and contextual realism by employing an immersive virtual environment featuring human avatars that approximate real-world encounters in terms of probability and plausibility. Within this setting, cognitive load during face perception and the mnemonic mechanisms supporting subsequent face recognition, as reflected in electrophysiological measures, are directly compared for dynamic 3D avatars and static 2D images of avatars. The results of these studies are discussed in terms of how the perceptual and conceptual processing of faces emerges for realistic faces in contrast to the conventional laboratory faces. This includes several stages of face perception, broader cognitive and attentional functions engaged during face processing, mnemonic mechanisms underlying face recognition, and the ways in which these processes are shaped by perceptual and contextual realism. In this context, the dissertation also examines the extent to which the field’s established reliance on individual electrophysiological markers of face processing can be meaningfully transferred to real-world situations, and proposes the development of an alternative, more comprehensive index-based approach. Additionally, the role of social VR in engram formation is considered, with a particular focus on how social engagement may enhance memory performance. Finally, the implications of the fundamental differences between realistic and laboratory-based face processing, spanning early perceptual mechanisms through later recognition stages, as well as differences arising from perceptual and contextual realism, are outlined with regard to future research directions.