I’ve Got Proof! Dataset-Specific Watermarking for Detecting Excessive Dataset Usage in Text-to-Image Diffusion Model Fine-Tuning

Currently, text-to-image diffusion models, which exhibit remarkable proficiency in image generation, have prompted the emergence of diverse fine-tuning methodologies due to the considerable resource demands entailed in their training. Simultaneously, apprehensions have arisen regarding the excessive utilization of open-source datasets for fine-tuning the models. Therefore, intellectual property protection for such datasets is important and necessary. In this regard, watermarking is a commonly employed method. However, the existing watermarking methods fail to establish an accurate correspondence between the watermarked dataset and the malicious model, which may lead to erroneous accusations against the model. In this paper, we propose a dataset-specific watermarking method, DSW, to detect excessive usage of protected datasets by suspect diffusion models fine-tuned using LoRA. DSW can establish a one-to-one correspondence between the watermarked dataset and the malicious model, thereby enhancing the accuracy of intellectual property verification. We simultaneously train an encoder and a decoder based on a bi-level optimization strategy. The encoder embeds a watermark image, which symbolizes the dataset owner's identity, into the dataset samples to yield a watermarked dataset. For a text-to-image diffusion model fine-tuned on this watermarked dataset, the decoder extracts the watermark from the generated image. Comprehensive experiments validate the availability, effectiveness, and stealthiness of DSW. The code is available athttps://github.com/hzhwis/DSW.