AI-enhanced adaptive testing with cognitive diagnostic feedback and its association with performance in undergraduate surgical education: a pilot study

Background Effective feedback in the cognitive domain is essential for surgical education but often limited by resource constraints and traditional assessment formats. Artificial Intelligence (AI) has emerged as a catalyst for innovation, enabling automated feedback, real-time cognitive diagnostics, and scalable item generation, thereby transforming how future surgeons learn and are assessed. Methods An item bank of 150 multiple-choice questions was developed using AI-assisted item generation and difficulty estimation. A formative Computerized Adaptive Testing (CAT), balanced across three cognitive domains (memory, analysis, and decision) and surgical topics, was delivered via QuizOne ® 3–5 days before the summative Progress Test. A total of 147 students participated, of whom 116 completed the formative CAT. Performance correlations, group comparisons, analysis of covariance (ANCOVA), and regression analyses were conducted. Results Students who voluntarily completed CAT showed higher Progress Test scores, though causality cannot be established due to self-selection bias ( p = 0.021), with the effect persisting after adjusting for prior academic performance (ANCOVA p = 0.041). Memory skills were the strongest predictors of summative outcomes ( R 2 = 0.180, β = 0.425), followed by analysis ( R 2 = 0.080, β = 0.283); decision was not significant ( R 2 = 0.029, β = 0.170). Conclusion AI-enhanced CAT–Cognitive Diagnostic Modeling (CDM) represents a promising formative approach in undergraduate surgical education, being associated with higher summative performance and providing individualized diagnostic feedback. Refining feedback presentation and enhancing decision-making assessment could further optimize its educational impact.