Artificial intelligence and human assessment consistency and bias in higher education academic writing evaluation

Abstract Artificial intelligence (AI) has widely influenced the practical activities of academic writing among current college students and has also provided an important transformation for academic writing assessment in higher education. However, many studies use English as the assessment background, and there is still room for further exploration on how teachers can use AI for assessment considering other culture background. This study explores the role of artificial intelligence in evaluating academic assignments, with a focus on the differences between AI assessment and teacher assessment in various dimensions of academic writing. The study employed purposeful sampling methods and utilized data from 92 undergraduate students majoring in educational technology. It adopted a five-dimensional evaluation framework, revealing the consistency of artificial intelligence with human evaluators in several key dimensions, particularly in logical structure. However, when evaluating research reviews and topics, there are differences. Artificial intelligence often overestimates performance, which may reflect AI bias. Further interviews and analyses with teachers and students revealed that the use of AI for evaluation was well-received by teachers, but concerns about reliability, trust, and privacy during the usage process persisted. The research proposes to establish a human-machine collaborative academic writing assessment method and adopt multiple approaches in teacher training and practice. This study underscores AI’s potential to revolutionize academic assessment. It provides a foundation for future research and practice for academic writing evaluation in higher education.