Can Software Developers Distinguish AI-Generated Code from Human-Written Code? A Preregistered Confirmatory Experiment

This study tests whether software developers can reliably distinguish AI-generated code from human-written code in a controlled experimental setting. Participants are shown code snippets, some generated by AI, some written by humans, and asked to judge whether each snippet is AI-generated, along with their confidence and perceived effort level of the code. This experiment serves as a validation study for a larger observational research project that uses an automated AI code detection algorithm to classify code contributions in open-source repositories and examine how AI-generated code is treated in code review. A key assumption of that research is that AI-generated code contains detectable features i.e. features that both automated tools and human reviewers can perceive. This experiment directly tests that assumption by measuring human detection ability in a controlled setting.