Expectation management in humans and LLMs

Abstract Mirative markers, such as “surprisingly”, explicitly encode a violation of expectations. Such markers are used for expectation management during communication. Sensitivity to mirative markers relies on two abilities: i) updating expectations upon recognizing a mirative marker, and ii) identifying expectation violations warranting the use of a mirative marker. In this study, we compared sensitivity to mirative markers in humans and large language models (LLMs). In part 1, we used a sentence-completion task, where humans and LLMs were presented with sentence fragments and asked to continue them. Results show that for both humans and LLMs, the presence of a mirative marker significantly increased response entropy and decreased top-response probability, in line with theoretical accounts of mirative processing. In part 2, we created a novel task, mirative polarity selection, where humans and LLMs are presented with a sentence pair and asked to select whether it was connected by a mirative marker (“surprisingly”) or an anti-mirative marker (“unsurprisingly”). Results show that LLMs perform at an impressive human-equivalent level. We conclude that both humans and LLMs use mirative markers as cues for calibrating their subsequent expectations during sentence processing.