Automated Detection of Invalid Responses to Creativity Assessments

Participants in creativity studies sometimes produce invalid data that is unusable for analysis,such as nonsensical or incomplete responses to idea generation tasks. Identifying such responsesis a time-consuming yet necessary process to ensure robust results but also remains challengingto automate. We explore the efficacy of transformer language models (TLMs) for automaticallydetecting invalid creativity responses, using only the text prompt and response and no othermetadata about the experimental session. We train a suite of transformers to detect invalid datafor two creativity assessments: the alternate uses task (AUT) and a design problems task (DPT).We compare TLMs against zero and few-shot large language models (LLMs) and other machinelearning models trained using word and sentence embeddings. We find that TLMs and LLMsgenerally outperform other baselines for both tasks. Further, we show that TLMs’ predictions arewell calibrated with respect to the quality of the participant response, ensuring that modelfailures will occur in a predictable way and that high-quality responses are unlikely to be labeledinvalid. Finally, we assess how well our models perform under an adversarial setting—using aprotocol where participants attempt to “break” the model by coming up with invalid responsesthat are nonetheless labeled valid. Our results demonstrate the potential of deep learning methodsfor cleaning creativity assessment data.