Comparing AI-Assisted Coding and Traditional Qualitative Analysis: a Study Examining Differences in Methods and Results of AI-Assisted Coding and Traditional Content Coding Using Community Engagement Data Collected During the Development of a Municipal Food Plan

Abstract Qualitative data poses a challenge for prevention science and public health, as it is critical to explain the context of communities, health, and behavior, yet collecting and analyzing qualitative data using traditional methods is time-intensive and requires extensive training. As artificial intelligence (AI) models have improved, there is a growing interest in using AI to code qualitative data quickly and reliably. This study compares the similarities and differences in methods and results of artificial intelligence (AI)-assisted qualitative analysis to traditional qualitative content analysis using data collected during the development of a city and county-based food plan. In total, 2820 community comments were collected across 43 community events in 27 zip codes across the region between March 2023 and January 2024. AI-assisted analysis was completed using a combination of a transcription app (Post-It Ⓡ ), GPT4 Plus, and GPT for Sheets with oversight from a public health practitioner. Traditional qualitative content analysis was completed with two trained coders who completed codebook development, reliability analysis, and full content coding. Both methods used deductive codes to represent key aspects of the food system and generated inductive codes to represent areas not included by the deductive food system codes. Results found that AI-assisted methods and traditional content analysis produced similar deductive coding results, while inductive coding results were less comparable across methods. Given that qualitative data has become a central part of prevention science, we believe with careful considerations, AI-assisted methods with intentional oversight have the potential to strengthen our ability to process large amounts of qualitative data.