Associations of adolescents’ argumentative writing scores and growth when evaluated by different human raters and artificial intelligence models

Having accurate and timely information on students’ argumentative writing skills is necessary to foster their improvement. Therefore, this study explored the associations among two different human rater types (researchers and teachers) and four different artificial intelligence (AI) models when using an analytic rubric to score eighth and ninth graders' argumentative writing gathered at two time points. Results revealed that, at each time point, AI models and teacher raters generally assigned higher scores than researchers. There tended to be moderate agreement of AI model scores with researcher scores that generally exceeded the agreement of teacher scores with researchers. However, there also tended to be greater agreement among the scores from AI models compared to the agreement of AI scores with either human rater type. There were some differences by writing trait and by AI model, suggesting ChatGPT-4o scores had the strongest association with researcher scores particularly for the task and development traits. When examining growth over time, the teacher and researcher scores demonstrated positive growth, whereas the AI scores generally suggested students declined. Again, the AI models had more concordance to each other than to human estimations of growth, but there was slightly more agreement in the estimates when comparing researchers to AI than to teachers. Implications for data use and interpretation are discussed.