DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion

Abstract Effective document intelligence models rely on large amounts of annotated training data. However,procuring sufficient and high-quality data poses significant challenges due to the labor-intensive andcostly nature of data acquisition. Additionally, leveraging language models to annotate real documents raises concerns about data privacy. In this context, synthetic document generation has emergedas a promising, privacy-preserving alternative. We propose DocDjinn, a novel framework for controllable synthetic document generation using Vision-Language Models (VLMs) that produces annotated documents from unlabeled seed samples. Our approach generates visually plausible and semantically consistent synthetic documents that follow the distribution of an existing source dataset through clustering-based seed selection with parameterized sampling. By enriching documents with realistic diffusion-based handwriting and contextual visual elements via semantic-visual decoupling, we generate diverse, high-quality annotated synthetic documents. We evaluate DocDjinn as a source of trainingdata across eleven benchmarks spanning key information extraction, question answering, documentclassification, and document layout analysis. Our experiments show that with only 100 real training samples, our framework achieves on average 87% of the performance of the full real-world dataset. To our knowledge, this is the first work demonstrating that VLMs can generate faithful annotated document datasets at scale from unlabeled seeds that can effectively enrich or approximate real, manually annotated data for diverse document understanding tasks. We publicly release our code and 140k+synthetic document samples.