The evaluator effect: Reliability challenges in AI-assisted assessment for business and educational technology

Read full text Summarise with VIRAT
Authors: Francis Melaragni, Phyllis Baron
Cited by: 0
References: 14

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

As educational institutions increasingly adopt large language models (LLMs) for automated assessment, grading support, and instructional feedback, fundamental questions emerge about the reliability of these systems. This study investigates whether LLM evaluators reach consistent conclusions when assessing identical student-like outputs—a critical validity concern for evidence-based educational practice. Five leading LLMs generated responses to seven domain-diverse prompts simulating typical educational tasks. Two LLM evaluators (Claude Sonnet 4 and ChatGPT-4o) independently assessed each response using standardized rubric criteria common in higher education. Results revealed systematic instability: Claude Sonnet 4's performance rating varied from 22.9% to 74.3% of criterion wins depending solely on which AI system conducted the evaluation—a 51.4 percentage point swing. This evaluator-dependent variance produced near-complete rank reversals, raising serious questions about the validity of AI-assisted assessment in educational contexts. The findings have immediate implications for: (1) faculty adopting AI grading tools, (2) institutions procuring educational technology, (3) researchers using AI evaluators in learning analytics, and (4) administrators developing AI governance policies. We argue that educational technology stakeholders must implement multi-evaluator validation protocols, transparent reliability reporting, and careful pilot testing before deploying AI assessment systems. This study contributes to the growing literature on responsible AI integration in teaching and learning while highlighting both opportunities and risks for educational innovation.

Fields of Study & Subjects

Artificial Intelligence in Education Student Assessment and Feedback Educational Technology and Assessment

Publication details

Journal / source
World Journal of Advanced Engineering Technology and Sciences
Volume
18
Issue
3
Pages
497–514
Publication date
31 March 2026
Language
EN