A method for evaluating diagnostic effectiveness using algorithms based on opinion obtained from artificial intelligence models

Objective : To develop a method for evaluating the diagnostic efficacy of differentiated algorithms based on artificial intelligence (AI) model output, as compared to conventional diagnostics. Material and methods. Two routing scenarios for diagnosing malignant and benign skin neoplasms were simulated. The first scenario involves using algorithms that rely on the output from the Derma Onko Check AI model. The second scenario involves standard patient routing to general practitioners, therapists, and dermatologists/venereologists for establishing a diagnosis. The used diagnostic efficacy indicators of algorithms based on output from the Derma Onko Check AI model as well as on reports from general practitioners, therapists, and dermatologists/venereologists were obtained from previous clinical studies. The modeling was conducted using the clinical data and photographic images of 90 patients with malignant skin neoplasms (39 melanomas and 51 basal cell carcinomas) and 291 patients with benign skin neoplasms (100 non-melanocytic skin tumors and 191 melanocytic skin tumors). Results . In order to evaluate the relative diagnostic efficacy of algorithms that rely on AI model output, calculation formulas were proposed, with visualization of the results in the form of a quadrant matrix. A mobile app called “AI-diagnostic efficiency calculator” (CalcRDAI&RNDAI) was developed for practical use to automatically compute the diagnostic diagnostic efficacy indicators of algorithms based on AI model output. Testing of the method to evaluate algorithms based on Derma Onko Check output reveals a 1.9-fold increase in the detection of skin cancer cases and a 10.5-fold decrease in missed cases. The evaluation results are in quadrant I (more cases are detected and fewer cases are missed), confirming the value of the diagnostic algorithm using algorithms relying on the Derma Onko Check AI model in the provision of medical care. Conclusion . The proposed method for evaluating diagnostic efficacy with the use of developed formulas and with the visualization of the results in the form of a quadrant matrix enables objective efficacy evaluation of AI models in multi-stage diagnostic routing.