Evaluating Creativity of LLMs through Video Games

This research project investigates whether Large Language Models (LLMs) can produce creative outputs as judged by human evaluators, using the complex, multi-stage task of designing and building a video game. By tasking proprietary models with generating unconventional game concepts, requiring a structured self-evaluation phase to approximate recursive creative thinking, and using coding agents for implementation and revision, this study takes advantage of LLMs' relatively newer ability to not just generate ideas, but implement them with a higher degree of autonomy. The project employs a mixed-methods evaluation through Prolific participants to assess the video games for creativity and related dimensions including originality, effectiveness, and perceived authenticity, while also utilizing an AI-aware vs. AI-blind disclosure condition to explore how human perception of creativity shifts when the producer is known to be artificial. Exploratory analyses evaluate LLM reasoning traces and intermediate outputs to assess the process of video game generation. In operationalizing creativity as human judgment, this project also highlights longstanding tensions around how creativity is defined and measured, and how they are sharpened when the creator is non-human. Expected outcomes include empirical insights into whether humans attribute creativity to LLM-generated video games, and how these attributions differ based on awareness of AI involvement.