Can Large Language Models Grade English Essays? A 19-Model Benchmark of Automated Essay Scoring Accuracy, Stability, and Prompt Sensitivity

This study benchmarks the essay scoring capabilities of 19 large language models (LLMs), comparing their accuracy against human expert ratings on standardized English writing assessments. We examine how prompt engineering strategies affect scoring accuracy and whether LLM scores demonstrate adequate test-retest stability for practical deployment in educational settings. Research Questions: - RQ1a: How accurately do LLMs score EFL essays compared to human expert ratings? - RQ1b: Does prompt engineering strategy (holistic → rubric → detailed → chain-of-thought) systematically improve scoring accuracy? - RQ1c: How stable (test-retest reliable) are LLM scores across repeated assessments? Hypotheses: - H1: Rubric-guided scoring (rubric/detailed/cot) yields higher human–LLM agreement (QWK) than holistic scoring, because structured prompts constrain the output space. - H2: The chain-of-thought (CoT) condition produces the highest QWK with human scores, as explicit reasoning before scoring reduces snap judgements. - H3: Larger models (70B+, GPT-4o, Claude Sonnet) outperform smaller models (8B–14B) on scoring accuracy. - H4: All models show ICC > 0.7 (good stability) at temperature 0.3. Failure Criteria: If QWK(LLM, human) < 0.4 for ALL models on the primary endpoint (overall score), we conclude LLMs are not yet suitable for automated EFL scoring.