Benchmarking Large Language Models on Real-Effort Tasks

Read full text Summarise with VIRAT
Authors: Francesco Fallucchi, Stefano Coniglio, Antonio Cosma, Federico Belotti
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Real-effort tasks are widely used in experimental economics to measure effort provision, cognitive performance, and responsiveness to incentives. A key implicit assumption underlying their use is that the decision-maker is human and therefore subject to effort costs, cognitive constraints, and motivational trade-offs. However, the increasing availability and performance of large language models (LLMs) raises a methodological concern. If LLMs can successfully complete canonical real-effort tasks at high levels of accuracy, then these tasks may no longer reliably measure human effort in contexts where the researcher cannot be certain that participants are human (e.g., online labor markets, open online experiments, unsupervised survey environments). In such cases, real-effort tasks could be completed by automated systems at low marginal cost, undermining their interpretability as measures of costly effort. This study evaluates whether commonly used real-effort tasks in experimental economics can be effectively solved by contemporary LLMs. Specifically, we (i) compare accuracy across different LLMs, (ii) examine how performance evolves across versions within the same model family, and (iii) test whether explicitly framing accuracy as monetarily incentivized affects LLM performance. The results inform the robustness of real-effort tasks as measurement tools in environments where participant identity cannot be fully verified. RQ1 (Cross-model accuracy): How does accuracy differ across LLMs on a set of established real-effort tasks? RQ2 (Version evolution): How does accuracy change across different versions of the same LLM family (e.g., ChatGPT versions)? RQ3 (Incentive framing): Does instructing the LLM about monetary incentives tied to accuracy affect accuracy? RQ4 (Economical): Is the use of LLMs instead of humans economically convenient?

Fields of Study & Subjects

Experimental Behavioral Economics Studies Computational and Text Analysis Methods Artificial Intelligence Applications

Publication details

Publication date
1 January 2026
Language
EN
Document type
Preprint