Data set creation for supervised deep learning–based analysis of microscopic images: Review of important considerations and recommendations

Supervised deep learning (DL) receives great interest for automated analysis of microscopic images with an increasing body of literature supporting its potential. The development and testing of those DL models rely heavily on the availability of high-quality, large-scale data sets. However, creating such data sets is a complex and resource-intensive process, often hindered by challenges such as time constraints, domain variability, and risks of bias in image collection and label creation. This review provides a comprehensive guide to the critical steps in data set creation, including (1) image acquisition, (2) selection of annotation software, and (3) annotation creation. For image acquisition, besides ensuring a sufficiently large number, it is important to address sources of image variability (domain shifts), such as those related to slide preparation and digitization, that could lead to algorithmic errors if not adequately represented in the training data. For annotations, key quality criteria are the 3 "C"s: correctness, consistency, and completeness. For mitigation of annotation bias of a single annotator, this review explores advanced annotation methods (eg, computer-assisted annotations). To support data set creators, a standard operating procedure is provided as supplemental material, summarizing all important considerations for data set creation. Furthermore, this article underscores the importance of open data sets in driving innovation and enhancing reproducibility of DL research. By addressing the challenges and offering practical recommendations, this review aims to advance the creation and availability of high-quality, large-scale data sets, ultimately contributing to the development of generalizable and robust DL models for pathology applications.