An evaluation of estimative uncertainty in large language models

Words of estimative probability (WEPs), such as “maybe” or “probably not” are ubiquitous in natural language for communicating estimative uncertainty. In linguistics, WEPs are hypothesized to have special (probabilistic) semantics, and their calibration with numerical estimates has long been an area of study. Motivated by increasing usage of large language models (LLMs) in applications requiring robust communication of uncertainty, this article studies how divergences in interpreting WEP between humans and LLMs reveal the limits of statistical language models in reproducing the subtleties of communication under uncertainty. Through a detailed empirical study, we show that established LLMs align with human estimates from an established (Fagen–Ulmschneider) survey only for some WEPs presented in English. Divergence is also observed for prompts using gendered and Chinese contexts. Upon further investigating the ability of GPT-4 to consistently map statistical expressions of uncertainty to appropriate WEPs, we observe significant performance gaps. The results contribute to a growing body of research on using LLMs to study complex communicative phenomena under diverse experimental conditions.