Scalable Distributed Training of Deep Learning Models for Medical Image Analysis: An Evaluation of Data Parallelism and Model Parallelism Strategies

Deep learning models have demonstrated significant promise in medical image analysis, but their training often requires substantial computational resources and large datasets. This paper investigates the performance of data parallelism and model parallelism strategies for scaling the training of deep convolutional neural networks (CNNs) on high-performance computing (HPC) clusters. We evaluate the scalability and efficiency of these approaches using synthetic medical imaging datasets, considering factors such as communication overhead, memory limitations, and the impact of data annotation quality. Our results highlight the trade-offs between these parallelization techniques and provide insights into optimizing distributed training workflows for medical imaging applications, acknowledging the importance of robust annotation practices, as highlighted by Gaggi (2026).