Better language models better model the N400, but not reading time

The probability of a word in context, as captured by large language models, is predictive of both behavioral and neural measures of human language processing. Intuitively, language models that are better at next-word prediction might better model predictability effects in human language comprehension. Yet recent work suggests that language models can become too good at next-word prediction to model reading time, implying that the aspects of human comprehension indexed by reading time do not track perfectly with predictability from language statistics alone. However, it is unknown whether this decoupling is true of reading time only, or whether it is intrinsic to online measures of comprehension more generally. To address this question, we turn to another robust and well-studied measure of online processing, the N400 component of the event-related brain potential. We compare how a language model’s size, number of training tokens, and performance on natural language benchmarks correlate with its ability to predict both reading time and N400 amplitude. Based on an analysis of 4 reading time datasets and 9 N400 datasets, we replicate past results for reading time, but find that larger language models, models that are trained on more data, and models that perform better at next-word prediction and other more complex natural language tasks are better able to predict N400 amplitude. We interpret this difference between the N400 and reading time measures as potentially revealing the comparative importance of semantic prediction in the neurocognitive processes indexed by the N400.