Comparing artificial intelligence and healthcare professional performance in surgical and interventional video analysis: a systematic review and meta-analysis

This systematic review and meta-analysis examines the design of studies comparing the performance of artificial intelligence (AI) with that of healthcare professionals in the analysis of videos from surgical and interventional procedures, and quantitatively evaluates the performance of AI, unassisted healthcare professionals, and AI-assisted healthcare professionals. From the 37,956 studies identified, 146 were included, with 76 providing sufficient information for inclusion in our exploratory meta-analysis. AI had significantly greater sensitivity and comparable specificity compared to unassisted healthcare professionals at their respective peak performance levels, with a relative risk of 1.12 (95% CI 1.07-1.19, p < 0.001) and 1.04 (95% CI 0.98-1.10, p = 0.224), respectively. AI-assisted healthcare professionals had significantly greater sensitivity and specificity compared to unassisted healthcare professionals across all levels of expertise, with a relative risk of 1.18 (95% CI 1.12-1.25, p < 0.001) and 1.05 (95% CI 1.02-1.08, p < 0.001), respectively. There was no significant difference in sensitivity and specificity of AI-assisted expert healthcare professionals versus AI, with a relative risk of 0.99 (95% CI 0.95-1.04, p = 0.787) and 1.03 (95% CI 0.97-1.08, p = 0.395), respectively. Whilst most studies to date have evaluated AI head-to-head against unassisted healthcare professionals, fewer studies examined AI as an assistive tool, despite the real-world integration of AI more likely to involve assistance than autonomy.