Situation Awareness Assessment for Anesthesia Residents (SAAAR): Development and Preliminary Evaluation of a Multimodal System

ObjectiveThis study reports the development and preliminary evaluation of SAAAR, a multimodal system designed to assess and support the development of situation awareness (SA).BackgroundSA is critical in anesthesiology, yet existing assessment methods lack standardized tools tailored to its complexities of anesthetic practice. Systems developed in other domains have limited applicability, highlighting the need for a purpose-built approach for anesthesia residents.MethodThe SAAAR comprises two components: a 16-item behavioral marker scale and a structured debriefing with eye-tracking. Thirteen anesthesiology faculty tested interrater and test-retest reliability, while five experts conducted content validation of the scale. Both components were implemented in a simulation-based training program for preliminary system evaluation.ResultsThe behavioral marker scale demonstrated moderate content validity and high reliability. Internal consistency was strong (McDonald's Ω = 0.928), test-retest reliability high (Spearman's ρ = 0.952), and interrater agreement moderate (Kendall's W = 0.412). Faculty reported the scale to be clear, comprehensive, and easy to use. Pilot implementation showed significant improvements across domains (Wilcoxon signed-rank test), indicating the system's potential to provide targeted feedback and guide educational interventions.ConclusionsGrounded in HFE principles, the SAAAR provides a structured approach to assessing SA in anesthesia residents and demonstrates preliminary potential to inform educational strategies. Further research is required to determine its impact on clinical performance.ApplicationThe SAAAR offers residency programs and human factors experts a practical tool for assessing SA and designing targeted training. Its adaptable framework suggests potential applicability in other high-pressure medical contexts, pending further evaluation.