Advanced Attention Checks: Using large language models to assess participant compliance in online studies

Online recruitment provides large and diverse samples but raises concerns about inattentive or non-compliant participants. Traditional attention checks (e.g., catch trials, reading checks) mitigate this issue but can be anticipated, bypassed, or costly to implement at scale. We introduce Advanced Attention Checks (AAC), a method that leverages large language models (LLMs) to automatically assess whether participants’ open-text responses meaningfully follow task instructions. In preregistered analyses of a large, multilingual dataset (N = 15,208), AAC demonstrated excellent test–retest reliability (φ = .95), substantial agreement with human coders (κ = .68), and modest convergence with traditional checks (κ = .22). Filtering with AAC amplified intervention effects in three of four outcome domains, though unexpectedly attenuated one effect. Confidence intervals became slightly wider, likely due to reduced sample size. Exploratory analyses further linked AAC compliance to greater effort and engagement. These findings position AAC as a scalable, cost-effective complement to traditional checks, supporting improved data quality in online studies.