Discourse Context and Co-Speech Gestures Jointly Shape Hierarchical Prediction During the Processing of a Multimodal Narrative

Understanding one another in daily communication depends on predicting language from prior discourse context and visual signals, such as co-speech gestures. However, it remains unclear how discourse context and gestures jointly shape neural predictions during naturalistic language processing. Here, participants watched multimodal narratives with spontaneously produced gestures during fMRI scanning. Leveraging transformer-based computational modeling, we disentangled linguistic predictability from contextual informativity at the sentence level, and observed that these deconstructed measures engaged neural regions associated with predictive processing across multiple multimodal representational levels. Furthermore, greater gesture availability was associated with attenuated predictability-related responses and weaker context-related effects, revealing a push-pull synergy between gestures and context. Our findings extend hierarchical predictive processing frameworks, demonstrating that gestures and discourse jointly, rather than additively, constrain neural predictions at multiple representational scales. These results demonstrate the critical and dynamically integrated role of multimodal predictive mechanisms in everyday communication.