From subtle breaks to missed diagnoses: Real-world evaluation of an artificial intelligence fracture detection tool

BACKGROUND Artificial intelligence (AI) shows promise in musculoskeletal imaging, particularly in detecting fractures in emergency situations. Nonetheless, doubts persist regarding its diagnostic precision, common error types, and the clinical impact of false-negative (FN) and false-positive (FP) outcomes. AIM To evaluate the AI fracture detection system's diagnostic performance vs radiology reports and analyse the causes and outcomes of FP and FN cases. METHODS We retrospectively reviewed radiographic examinations over 3 months interpreted by both AI and radiologists. Radiology reports served as the standard of reference. Diagnostic accuracy metrics were calculated from the primary dataset. FP and FN cases were further examined using dedicated outcome sheets with explanatory notes categorised into predefined reason groups, and clinical outcomes classified into standardised categories. RESULTS The AI algorithm achieved an accuracy of 94.0%, a sensitivity of 89.9%, a specificity of 96.1%, a negative predictive value of 96.4%, and a positive predictive value of 89.8%. Out of 563 fractures identified in radiology reports, 54 (9.6%) were missed by AI. Most false negatives (92.6%) were due to subtle or minimally displaced fractures, primarily at the wrist and ankle. False positives (n = 58) were mainly caused by degenerative changes, growth plate variations, or healed fractures (91.4%). FN cases had greater clinical impact, often requiring fracture clinic follow-up (61.1%), additional imaging (14.8%), or hospital admission (5.6%), whereas FP cases mainly led to unnecessary follow-up or imaging. CONCLUSION AI demonstrated strong diagnostic performance, comparable to that in the published literature, but remains limited in detecting subtle and anatomically complex fractures. FN errors pose risks of delayed or missed treatment, while FP errors increase resource utilisation. AI should be integrated as a triage and decision-support tool with radiologist oversight, and future refinement should target wrist and ankle injuries and better differentiation of chronic from acute findings.