A Linguistic Analysis of Human and LLM Annotations on English Sentence Difficulty

English language education is one of the most widespread learning domains worldwide, and effective instruction requires personalization aligned with learner proficiency. A critical component of personalization is the precise labeling of text difficulty. While human/expert-based approaches and LLM-based methods are commonly used, systematic analyses of their frequent discrepancies remain limited. This study compares CEFR-labeled datasets annotated by experts and state-of-the-art LLMs, examining the influence of linguistic factors such as sentence length, vocabulary diversity, readability, and syntactic complexity. Results show that average sentence length and the Gunning Fog Index best explain human–model disagreement, with most models more prone to errors on short and simple sentences. Moreover, ensembles including GPT-5.1-mini improved performance, whereas those including LLaMA-4 degraded it. These findings suggest that LLMs can approach human-level annotation under certain conditions, and that discrepancy analysis offers practical value for label validation and model selection.