Image Diffusion Models With Multimodal Conditional Control in Zero-Shot Semantic Segmentation Synthesis

In the era of booming large models, the significance of data scale in deep learning has been well-acknowledged. Nevertheless, obtaining large-scale datasets remains a challenging task. Large pre-trained diffusion models, boasting remarkable generative capabilities, offer a promising solution for dataset generation. This study zeroes in on the labor-intensive semantic segmentation task, where image annotation is a major bottleneck. We introduce a novel data generation pipeline leveraging multimodal conditional control. This pipeline not only enhances the universality and practicality of our method but also ensures high-precision alignment between generated images and corresponding masks. Specifically, we feed mask, Canny edge, and depth information into ControlNet and employ captions generated by the BLIP model as prompts to precisely control image generation. Notably, we pioneer the exploration of zero-shot generation in this context. This approach enables direct image generation without the need for fine-tuning or alignment with segmentation protocols. Experimental results demonstrate its effectiveness: in the CityScapes dataset, zero-shot generation leads to a 0.6% increase in the mean Intersection over Union (mIOU). Even without zero-shot generation, our method achieves significant mIOU improvements of up to 2.53% on the ADE20K dataset and 1.93% on the COCO-Stuff dataset. These findings highlight the potential of our proposed method in advancing semantic segmentation tasks.