Side- and patient-based performance of a deep learning system based on the results of individual detection of carotid artery calcifications on panoramic radiographs

Purpose: The present study aimed to develop 2 deep learning (DL) systems incorporating detection functions for the diagnosis of carotid artery calcifications (CACs) on panoramic radiographs and to compare their diagnostic performances using CAC-based, side-based, and patient-based evaluations. Materials and Methods: Panoramic radiographs from 290 patients with CACs and 290 control patients without CACs were used to develop 2 detection models: one designed to detect individual CACs across the entire radiograph (System 1) and another designed to detect CACs within the limited bilateral cervical areas (System 2). CAC-based performance was evaluated using recall, precision, and F1-score. Side-based and patient-based performances were assessed using sensitivity, specificity, positive predictive value, negative predictive value, accuracy, and the area under the receiver operating characteristic curve (AUC). Results: For System 1, CAC-based recall, precision, and F1-score were 0.81, 0.68, and 0.74, respectively. For System 2, the corresponding values were 0.90, 0.67, and 0.77. Side-based sensitivity, specificity, and AUC were 0.87, 0.80, and 0.83 for System 1, and 0.93, 0.84, and 0.89 for System 2. Patient-based sensitivity, specificity, and AUC were 0.93, 0.73, and 0.83 for System 1, and 0.95, 0.70, and 0.83 for System 2. Although a relatively large number of false positives were observed in CAC-based assessments, side-based and patient-based performances showed improvement. Conclusion: Side-based and patient-based performances were sufficient when calculated on the basis of CAC-based evaluations for diagnosing CACs on panoramic radiographs. When conducting studies of this type, performance assessments should include side-based and patient-based evaluations in addition to CAC-based analyses.