Evaluation of Moral Motivation Inference in Real-World Moral Narratives by Humans and Frontier Language Models

We will evaluate how moral motivations are inferred from natural-language descriptions of real-world moral narratives across two populations: human participants and frontier language models (LMs). Using a shared set of 344 moral narratives, both populations will annotate each narrative for the presence/absence and perceived strength of four primary moral motivations defined by the Motivation of Moral Action (MoMA) framework: Approach Good, Avoid Bad, Approach Bad, and Avoid Good. Subjects will only infer the presence of motivation based on the explicit presence of wording, not speculate about motivations not tied to actual text (i.e., this is how I would expect myself or others to feel in this situation, even if they didn’t say it). In addition, annotators will (a) identify the sentence(s) within each narrative that justify their motivational inferences, (b) classify the intended outcome of the agent’s action as morally good or bad, and (c) answer additional judgment questions. The primary goal of this study is to test whether the MoMA framework captures systematic patterns in how moral motivations are inferred from natural language and whether these patterns appear within each population independently. All hypotheses, measures, and analyses are identical for human and LM data and will be conducted separately for each population. No direct statistical comparisons between humans and LMs are planned for this preregistered study. For the LM condition, we will use a suite of frontier language models selected based on publicly available performance benchmarks (e.g., LM Arena rankings at the time of data collection), the country of origin, and the historical track record of the lab that produced the model. We use the term “language model (LM)” rather than “large language model (LLM)” to acknowledge potential architectural differences across systems and to avoid assuming model design homogeneity. LMs will be prompted to perform the same annotation tasks as human participants, producing structured outputs for motivation presence, motivation strength, justification sentences, and outcome classification. For the human condition, participants will be recruited online (e.g., via Prolific, Besample, etc.) and will complete the same annotation tasks using identical materials and response formats. Each participant will rate a subset of narratives (10), with the assignment structured to ensure adequate independent coverage of each narrative across raters. By applying the same theoretical framework and analytic pipeline to humans and LMs separately, this study provides a principled test of the MoMA model’s generality as a description of moral motivation inference from natural language.