Generative AI and Extension Program Planning and Evaluation: Do General Large Language Models Answer Domain-specific Questions Accurately?

Generative artificial intelligence (Gen AI) promises to revolutionize Extension program planning and evaluation (EPPE) by enabling Extension professionals to assess needs, design, and evaluate programs efficiently. However, research has yet to ascertain the accuracy of Gen AI, specifically Large Language Models (LLMs) like ChatGPT, in responding to EPPE-related prompts. This study employed expert judgment to assess the accuracy of ChatGPT responses to EPPE-specific prompts from the perspectives of two EPPE specialists, professors, and faculty who work with the United States Cooperative Extension Service (CES). Using frequencies, percentages, Cohen’s kappa, and inductive content categorization to analyze the data, the study showed that experts rated ChatGPT’s responses “partially correct” for 60% of the prompts, with the remaining responses rated either “correct” or “mixed.” None of the responses were rated as “irrelevant,” indicating that ChatGPT’s responses were consistently relevant for all the EPPE topics. However, the inter-rater agreement was low (Cohen’s k = .38, p=.025), revealing variability in expert judgment. Inaccuracies in ChatGPT’s responses resulted from a mismatch with technical evaluation standards and insufficient contextual information. In conclusion, ChatGPT demonstrates potential as a support tool for EPPE, however, expert oversight, responsible and ethical use, and a chatbot trained on research-based EPPE data could enhance its response accuracy and reliability. We recommend the implementation of Extension capacity-building initiatives that build professionals' capacity to use Gen AI responsibly and ethically, examine Gen AI’s responses critically, blend expertise with AI-generated responses, and write effective prompts (prompt engineering) to enhance Gen AI's potential utility in EPPE.