2106 Structural-Equation-Based Correction of Attending Rater Bias in Video-Based Assessment of Resident Craniotomy Performance: A Pilot Study

View at publisher Summarise with VIRAT
Authors: Jose Luis Porras, Melissa Canales, Zoe Soulé, Ariel Leong, Lily Fritz, Lalithkumar Seenivasan, Roger D. Soberanis-Mukul, Rohit Shankar, Brendan F. Judy, Nancy A. Abu-Bonsrah, Alice L. Hung, Andrew S. Luksik
+8 more Wataru Ishida, Risheng Xu, Jordina Rincón-Torroella, Gary L. Gallia, Judy Huang, Henry Brem, Mathias Unberath, Masaru Ishii
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

INTRODUCTION: Video assessment provides scalable, objective benchmarking of operative skill, a critical need as ACGME milestones and competency-based promotion demand defensible metrics. In neurosurgery, the fine motor precision required and the consequences of error mean that residents receive autonomy in much smaller, high-stakes increments than in most other fields, making truly accurate skill measurement indispensable. However, uncontrolled evaluator bias can distort scores and undermine the decision to entrust a trainee with advancing autonomy. We propose using structural equation modeling (SEM) to create an unbiased measure of resident skill by isolating rater effects during video assessment of neurosurgery. METHODS: Five de-identified craniotomy videos (three residents) were independently scored on an eight-item, OSATS-derived 100-point scale by six faculty evaluators (two attendings, four chiefs), resulting in 18 rating events. A two-factor SEM was fitted in Stata: Skill (resident ability) and Bias (evaluator leniency), with each variance fixed at 1 for identification. Standardized loadings and coefficients were tested using z-statistics. RESULTS: Skill loadings were robust across all items (λ = 0.63–0.89, p < 0.001). Four technical domains—Instrument Handling, Flow, Hemostasis, Non-Dominant Hand—showed significant rater bias (λ = 0.51–0.69, p = 0.041–0.010); global tissue-handling items did not (p > 0.30). Bias adjustment reduced item-level residual variance by up to 74% and decreased overall unexplained variance 36%. Bias-corrected composite scores ranked residents exactly as evaluators’ perceived postgraduate level. CONCLUSIONS: SEM isolated and corrected attending bias even in a sparse dataset, providing a practical route to fairer neurosurgical skills assessment. The approach supports milestone-based promotion, national benchmarking, and high-quality labels for forthcoming AI feedback tools. Ongoing institutional accrual will enable cut-score definition, but this proof-of-concept already merits a broader discussion of objective, scalable neurosurgeon-skill analytics.

Fields of Study & Subjects

Innovations in Medical Education Surgical Simulation and Training Psychometric Methodologies and Testing

Publication details

Journal / source
Neurosurgery
Volume
72
Issue
Supplement_1
Pages
273–273
Publication date
26 March 2026
Language
EN