Enhancing Automated Text Coding in Online Learning Research: A Systematic Calibration Framework for Large Language Models

Automated coding is particularly crucial in online learning research, where vast amounts of text provide valuable insights into cognitive engagement, emotional expression, and social interaction, yet manual analysis of such large-scale discourse remains time-consuming. While Large Language Models (LLMs) offer promising solutions, existing approaches suffer from inconsistent coding performance and suboptimal prompt engineering, limiting their reliability across diverse educational frameworks. This study introduces the Optimized Large Language Model Coding (OLLM-C) framework, a systematic approach that enhances automated content analysis through calibrated prompting strategies. We evaluated OLLM-C using a comprehensive dataset of 8,671 comments collected from five online courses. The framework was validated against 15 widely-used coding frameworks spanning cognitive, social-emotional, and behavioral dimensions. Comparative analysis of five LLMs (GPT-3.5, GPT-4o, Gemini, Claude, and Llama) revealed GPT-4o's superior performance in initial coding tasks. The OLLM-C calibration process significantly improved GPT-4o's reliability, with Cohen's Kappa coefficients increasing from an average of 0.45 to 0.57 across frameworks. Notably, the model demonstrated stronger performance in social-emotional coding compared to cognitive skills frameworks, achieving substantial agreement with human coders in emotion recognition and social interaction analysis while showing limitations in complex cognitive reasoning tasks. These findings establish OLLM-C as a systematic calibration framework that enhances the reliability, efficiency, and practical applicability of LLM-assisted qualitative analysis.