Integrating clinical reasoning into the accuracy and effectiveness evaluation of diagnostic artificial intelligence tools in dermato-oncology

Background. Standard error matrices systematically underestimate the clinical value of diagnostic artificial intelligence (AI) tools by misclassifying cautionary conclusions regarding histologically benign, yet clinically suspicious neoplasms with clinical signs of malignancy as false-positive outcomes. Objective: To develop and validate a methodology for assessing the clinical effectiveness and accuracy of diagnostic AI tools in dermato-oncology, incorporating clinical caution alertness as an independent metric. Material and methods. A total of 342 skin lesions were evaluated at the Burdenko Main Military Clinical Hospital (2025–2026). While a standard error matrix was used or the formal accuracy assessment, the clinical evaluation relied on two newly developed: a four-category clinical effectiveness system (complete concordance, concordance in clinical caution, discordance in clinical caution, complete discordance), and a clinical accuracy formula derived from a clinical case matrix (Justified Conclusion, Unjustified Conclusion, Missed Justified Conclusion and Not Missed Justified Conclusion). Wilson confidence intervals (CIs) were applied. Results. The formal evaluation yielded an accuracy was of 89.7% (100.0% sensitivity, 84.8% specificity). The analysis of clinical effectiveness revealed complete concordance in 82.2% of cases, concordance in clinical caution in 12.9%, discordance in clinical caution in 1.2%, and complete discordance in 3.8%. Clinical accuracy metrics were as follows: 98.5% sensitivity (95% CI 94.6–99.6), 88.1% specificity (95% CI 83.0–91.8), 92.1% accuracy (95% CI 88.8–94.5). Conclusion. Relying solely on the formal diagnostic accuracy of AI tools underestimates their clinical value. The proposed clinical accuracy and effectiveness metrics ensure an objective assessment of AI tools in relation to the real-world tasks in dermatooncological patient routing.