CAD score: context-aware semantic diversity for scalable creativity evaluation of LLMs

Read full text Summarise with VIRAT
Authors: Junsu Kim, Geunyeong Jeong, Harksoo Kim
Cited by: 0
References: 32

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Large Language Models (LLMs) are increasingly recognized for their potential in creative applications, yet quantitatively evaluating the diversity of their generated responses remains an underexplored and challenging problem. Beyond simple lexical variation, creative diversity in LLMs refers to the capacity to generate a wide spectrum of conceptually distinct and contextually relevant ideas in response to open-ended tasks. However, existing automated metrics often operate in a ‘context-free’ manner, disregarding the relationship with the prompt, or, like the recent ‘LLM-as-a-Judge’ approach, suffer from high costs and low reproducibility. To address these problems, this article proposes a new metric, the Context-Aware Diversity (CAD) score. CAD generates context-conditioned embeddings by explicitly combining each answer with the query, and then quantifies the semantic volume of the generated answer set by applying the eigenvalue entropy calculation from the Vendi score. We validated the CAD score on seven LLMs, spanning the Llama-3, Qwen-2.5, and Gemma-2 families, using a benchmark of seven creative tasks based on the Torrance Tests of Creative Thinking (TTCT). Validation results show that the CAD score demonstrates a consistently higher Spearman correlation with GPT-4-judged flexibility scores than all baselines, including surface-level and other context-free semantic metrics. Notably, in a partial correlation analysis controlling for the confounding variable of answer length, existing baselines failed to retain statistical significance, whereas CAD maintained a robust correlation. Moreover, an ablation version that intentionally removed context information also lost this statistical robustness, demonstrating that context-conditioning is essential for meaningful evaluation. Our further analyses also revealed that model scale does not necessarily correlate with diversity, that performance rankings vary depending on task type, and that models exhibit distinct generative tendencies, ranging from exploratory to conservative. These findings suggest that the CAD score is useful not only as a quantitative benchmark for creativity evaluation but also holds practical potential in applications such as a reward signal for post-training or for evaluating candidate responses in Retrieval-Augmented Generation (RAG) systems.

Fields of Study & Subjects

Topic Modeling Creativity in Education and Neuroscience Natural Language Processing Techniques

Publication details

Journal / source
PeerJ Computer Science
Volume
12
Pages
e3624–e3624
Publication date
27 March 2026
Language
EN