Evaluating Deep Learning for Clinical Decision-Making: A Case Study on Diabetic Retinopathy

Deep learning has shown promise in automating medical image analysis, particularly in predicting diabetic retinopathy (DR), a diabetes-related eye condition that can lead to blindness. The study "BRSET: A brazilian multilabel ophthalmological dataset of retina fundus photos" by Nakayama et al. (2024) reports strong performance (AUC-ROC = 0.97 for the ConvNeXtv2 model) using two convolution neural networks (CNNs) on retinal fundus images. However, critical limitations remain in the study, such as lack of reproducibility, reliance on data leakage datasets, lack of external validation, limited model architectures, and narrow evaluation metrics.This thesis addresses these gaps by investigating model reproducibility, data leakage, model architecture and size, data efficiency, and external validation. Recent advances in large foundation models have demonstrated strong generalization capabilities across domains, including medical imaging. In addition to CNN-basedmodels, the study evaluates vision transformer-based (ViTs) foundation models, including DINOv2, RetFound, and VisionFM. Evaluation metrics include conventional measures like macro AUC-ROC and F1-score, alongside clinically relevant tools such as calibration curves and the Polytomous Discrimination Index (PDI), which is an extension of AUC-ROC for multi-class classification. The results show that reproduced patient-stratified models has lower performance (AUC-ROC = 0.93 for the ConvNeXtv2 model). The larger ViT models do not outperform CNNs. In general, all models show poor calibration affects the reliability of the models. External validation further reveals challenges in generalizability (AUC-ROC = 0.56 for the ConvNeXtv2 model). These findings emphasize the need for rigorous, multifaceted evaluation in developing AI tools for clinical use and caution against over-interpreting results from studies that rely on limited validation or narrow evaluation criteria.