AI in the Experimental Loop: Implications for Replicability in Social Sciences
Summary
No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.
Summarise with VIRAT Free to try with your Science Catalogs account.Abstract
Integrating generative Artificial Intelligence (AI), in particular Large Language Models, into experimental social sciences presents both powerful opportunities and significant methodological challenges. One major issue is the stochastic nature of the output of those models, which complicates replicability—a foundational principle of scientific research. This paper examines the transition from the traditional Experiment-Subjects Dyad (ESD) experimental design to what we refer to as an Experiment-AI-Subjects Triad (EAIST) experimental design, where AI is employed to generate experimental trials. In the EAIST experimental design, AI can provide adaptive stimuli and generative systems, which may undermine experimental control and threaten the reliability of Human-Robot Interaction studies. We review specific examples, such as emotionally expressive chatbots and GAN-generated facial expressions, and identify the two sources of variability they introduce. We then propose a framework to enhance replicability in AI-driven research, drawing on principles from the Open Science movement. Key strategies include modular testing, parameter fixation, structured prompt engineering, and robust experimental design. We provide a checklist to ensure robustness and replicability of the EAIST experiments. Our recommendations aim to maintain the ecological advantages of AI while reinforcing methodological transparency and scientific rigour.
Fields of Study & Subjects
Ethics and Social Impacts of AI
Social Robot Interaction and HRI
Computational and Text Analysis Methods
Publication details
- Journal / source
- OSF Preprints (OSF Preprints)
- Publication date
- 31 March 2026
- Language
- EN
- Document type
- Preprint