Evaluating GPT as automated analyzer for detecting students’ erroneous mental models in programming education

Abstract Purpose This study evaluates Large Language Models as automated analyzers for detecting students’ erroneous mental models in programming education, addressing the scalability barrier of labor-intensive manual analysis. Building upon preliminary GPT-4 findings, this investigation extends to GPT-5, assessing generational improvements in reliability, error patterns, and computational feasibility. Methods GPT-4 and GPT-5 analyzed 1,500 student responses to 24 C++ programming questions, yielding 5,300 binary judgments for 27 hypothesized mental models compared against expert faculty evaluations. Inter-rater agreement was quantified through percentage agreement and Cohen’s Kappa, with confusion matrix analysis revealing systematic error patterns. Results GPT-5 achieved 82.7% raw agreement with faculty (vs. GPT-4’s 66.9%) and Cohen’s Kappa of $$\kappa $$ = 0.38 (vs. 0.18), though below the 0.60-−0.80 inter-human threshold. Both models exhibited false positive bias, with GPT-5’s reduced rate (11.6% vs. 23.1%) driving improvement. GPT-5 demonstrated 7.6-fold speed improvement and 34% cost reduction ($12.51 vs. $19.06). Conclusion Current LLMs achieve production-ready status within hybrid human-AI frameworks. GPT-5’s fair-to-moderate reliability combined with cost-performance improvements ($50 per 100-student cohort) establishes feasibility at educational scales. However, reliability insufficient for autonomous deployment supports hybrid configurations wherein LLMs serve as first-pass analyzers subject to systematic faculty review.