Patient-level evaluation of deep learning models for pre-operative diagnosis of adnexal masses on ultrasound imaging

For the implementation of artificial intelligence and machine learning (AI/ML) in medical imaging, algorithm performance is usually reported across all patients in summary figures of merit such as the area under the receiver operating characteristic curve (AUROC) or sensitivity and specificity at an operating point. Sometimes, multiple AI/ML models for a specific task have similar or comparable model performance, but it would be useful to evaluate whether their outputs are similar or different for an individual patient, especially in deep learning (DL) models. In particular, AI/ML models are of great interest in differentiating between malignant and benign adnexal masses on ultrasound, given the modality’s role in initial patient evaluation, and tools are needed beyond AUROC to evaluate model performance for specific patients in order to ensure clinical relevance and utility. We aimed to evaluate and compare the performance of a support vector machine (SVM) classifier at the patient-level in differentiating between benign and malignant adnexal masses on ultrasound imaging. Four variations of U-Net DL models were intentionally chosen to extract features from two design aspects common in DL approaches: (1) with ‘from scratch’ training or with transfer learning and (2) with or without inclusion of an attention mechanism. Our results showed while the AUROC was similar among all four models, the patient-level results varied widely: some cases were correctly classified by all four models, some were incorrectly classified by all models, and some cases had combinations of correctness. These results point to the importance of evaluating algorithm performance at the patient level, especially when the goal is to incorporate more advanced DL models for complex clinical tasks on medical images.