A Comprehensive Item Pretesting Study to Develop an ASAG Platform for Turkish Inputs

This study presents a comprehensive framework for pretesting ASAG items, addressing a critical gap in standardized item-development procedures in the automated-scoring literature. We utilized an LLM zero-shot with rubric scoring approach to develop and assess an ASAG platform for Turkish-language inputs across three subject areas: Turkish revolutionary history, Turkish as a first language, and science literacy. Our methodology integrated traditional classical test theory parameters with insights from cognitive interviews and expert-based machine score-ability ratings to predict ASAG performance. We evaluated performance using quadratic weighted kappa across 73 short-answer items. The results indicated that the effectiveness of the pretesting metrics varies by domain. While strong correlations between item parameters and ASAG performance were observed in the history and Turkish domains, a more complex case arose in the science literacy domain, where pretest metrics were insignificant predictors, likely due to linguistic diversity and multi-level reasoning. These findings suggest that automated scoring approaches better manage semantic equivalence and linguistic variation than expert predictions, especially in diverse domains such as science. Overall, this study offers suggestions for transitioning from retrospective performance analysis to a proactive item-development process for ASAG.