Generative deep learning for foundational video translation in ultrasound

For deep learning (DL) to realize its potential for medical image interpretation, attention to dataset content is critical. Ultrasound presents a particular challenge because, in addition to many views and structures, it includes several sub-modalities–such as grayscale and color flow doppler (CFD)–that are often imbalanced or confounding in clinical datasets. Image translation could help, but it has not yet succeeded in noisy ultrasound. Here, we develop generative video translation for CFD-to-grayscale ultrasound. We leveraged pixel-wise, adversarial, and perceptual losses to synthesize anatomically faithful, realistic-looking ultrasound. Average SSIM between synthetic and ground-truth videos was 0.91 ± 0.04. Synthetic videos performed indistinguishably in DL classification (F1-score between real and synthetic, 0.93–0.95) and segmentation tasks (average Dice between real and synthetic segmentations, 0.97 ± 0.03). Blinded clinician accuracy in distinguishing real vs. synthetic videos was 54 ± 6%, indicating realism. Although trained only on heart videos, the model worked on ultrasound spanning clinical domains (average SSIM 0.91 ± 0.05), demonstrating foundational abilities. Applying generative translation to real-world CFD imaging recovered over seven percent more data for a clinical DL task. Together, these data expand the utility of retrospective imaging, advance rigor in medical synthetic data evaluation, and augment the dataset design toolbox for medical imaging.