Neural integration of affective prosodic and semantic cues in non-literal forms of speech understanding

Abstract Emotions in speech are conveyed through both semantics and prosody—voice ‘melody’, yet how listeners integrate these cues in the brain remains unclear. We investigated non-literal forms of speech, such as irony and sarcasm, where understanding beyond literal meaning relies on the dynamic interplay between affective prosodic and semantic cues, alongside theory of mind (ToM). We used functional magnetic resonance imaging while participants listened to short dialogues between two characters, which varied in prosody and semantics to convey either literal or non-literal meanings. Behaviorally, semantics and prosody interacted in shaping participants’ evaluations, although the data suggest a prosody dominance effect. Non-literal speech engaged a distributed network including the bilateral inferior frontal gyrus, temporal speech regions, and ToM areas, with ROI analyses revealing heterogeneous prosody-semantics integration profiles across regions and tasks. Taken together, our data clarify the behavioral and neural underpinnings of the integration of prosody and semantics in non-literal speech and open new venues in this field. Author note The authors made the following contributions. Adrien Wittmann: Conceptualization, Methodology, Data curation, Formal analysis, Software, Investigation, Validation, Visualisation, Writing – Original Draft Preparation, Writing – Review & Editing; Leonardo Ceravolo: Methodology, Software, Writing – Review & Editing; Audrey Mayr: Conceptualization, Software, Methodology, Investigation, Validation; Didier Grandjean: Funding acquisition, Supervision, Conceptualization, Writing – Review & Editing.