Deep learning-based classification of thyroid nodules using uncertainty-aware multi-modal ultrasound imaging

Accurate differentiation of thyroid nodules is crucial for timely diagnosis of thyroid cancer. Most recent studies utilize grayscale ultrasound with deep learning to distinguish benign from malignant thyroid nodules. The goal of this study is to effectively boost the performance of classification of thyroid nodules using multi-modal ultrasound imaging combining B-mode, color Doppler (CD), and shear wave elastography (SWE) within a customized deep learning architecture. This study prospectively included 506 thyroid nodules acquired from 422 subjects. The proposed network integrated a pretrained MobileNetV2 backbone with a shallow head composed of depth-wise separable convolutional layers, attention-based mixed pooling, and a tailored self-attention mechanism-applied for the first time in the context of thyroid nodule classification. To further improve robustness, we introduced a patient-level uncertainty-aware fusion strategy that selectively integrates predictions from each modality based on their validation scheme and achieved a classification accuracy of 0.95, a sensitivity (Sen) of 0.98, an area under the ROC curve (AUC) of 0.97 (95% CI: 0.94-0.99), a specificity of 0.92, and an F1 score of 0.95, on test data. Multi-modal images improved the performance (AUC: 0.97) as compared to uni-modal (AUC range: 0.73-0.90) or bi-modal (AUC range: 0.90-0.97) data. Further comparative analysis showed that the proposed network performed similarly or better to the state-of-art deep learning networks (AUC range: 0.82 -0.97) for thyroid nodule classification while utilizing relatively smaller architecture. Finally, the integration of multi-modal ultrasound imaging with a customized deep learning network effectively and efficiently improved the overall classification of thyroid nodules, which can further enhance diagnostic performance.