Integration of spatiotemporal features into machine learning assessment of open surgical skills

INTRODUCTION: Accurate and objective assessment of operative skills is essential for improving training paradigms, patient safety, and quality of surgery. Advances in machine learning have facilitated automated assessment of minimally invasive and robotic operations. This study aims to develop a novel machine learning model for evaluation of open surgical proficiency. METHODS: This study used the AIxSuture data set. A global rating score was assigned for each video, categorizing individuals into novice (n = 119), intermediate (n = 79), and proficient (n = 116) classes. Hybrid convolutional neural network and long-short-term-memory networks were employed to train the video classifier model. ResNet50, an image classification model, served as a spatial feature extractor to perform nonlinear transformations. Long-short-term-memory networks selectively retained and discarded both significant and insignificant changes in frame sets that capture the subject's movements. The class-wise F1 score was measured to assess harmonic performance. RESULTS: Our assessment achieved a mean F1 score of 80.1% in determining the performance level of each subject, outperforming previous models. Additionally, the model classified performance with 90.1% accuracy for the novice group, 65.7% for the intermediate group, and 86.3% for the proficient group. Despite lower accuracy in the intermediate class, this metric outperformed other models in this group by nearly 10%. The present model classified each video into appropriate skill levels at an estimated 10.2 ± 0.4 seconds. CONCLUSIONS: Our machine learning model provides a robust framework for skill assessment in open surgery. The application of machine learning in clinical practice should be considered to evaluate surgeons' skills and help improve training and outcomes.