arXiv:2602.21824cs.LG2026-02被引 1

用视觉语言模型生成可控、高质合成文档,仅需少量真实数据即可达到接近全量真实数据的性能。

DocDjinn: Controllable Synthetic Document Generation with VLMs and Handwriting Diffusion

  • 基于聚类选择种子样本,结合参数化采样生成语义一致的合成文档。
  • 仅用100个真实样本即达全量数据87%性能,支持多种文档理解任务。
  • 适合需要隐私保护或数据稀缺场景的研究者与开发者使用。

高效文档智能模型依赖大量标注数据,但获取高质量数据成本高且耗时,使用语言模型标注真实文档又存在隐私风险。合成文档生成成为一种有前景的隐私保护替代方案。我们提出DocDjinn,一个利用视觉语言模型(VLMs)的可控合成文档生成框架,可从无标签种子样本生成带标注文档。通过基于聚类的种子选择与参数化采样,生成符合源数据分布的视觉逼真、语义一致的合成文档。借助语义-视觉解耦的扩散模型生成逼真手写体和上下文视觉元素,提升多样性与质量。在11个基准上评估,涵盖关键信息提取、问答、文档分类与版面分析。据我们所知,这是首个证明VLM能从无标签种子大规模生成忠实标注文档数据集的工作,可有效补充或近似真实标注数据。仅用100个真实训练样本,平均达到全量真实数据87%的性能。代码与140,000+合成文档样本已公开。

原文摘要 · Abstract (English)

Effective document intelligence models rely on large amounts of annotated training data. However, procuring sufficient and high-quality data poses significant challenges due to the labor-intensive and costly nature of data acquisition. Additionally, leveraging language models to annotate real documents raises concerns about data privacy. Synthetic document generation has emerged as a promising, privacy-preserving alternative. We propose DocDjinn, a novel framework for controllable synthetic document generation using Vision-Language Models (VLMs) that produces annotated documents from unlabeled seed samples. Our approach generates visually plausible and semantically consistent synthetic documents that follow the distribution of an existing source dataset through clustering-based seed selection with parametrized sampling. By enriching documents with realistic diffusion-based handwriting and contextual visual elements via semantic-visual decoupling, we generate diverse, high-quality annotated synthetic documents. We evaluate across eleven benchmarks spanning key information extraction, question answering, document classification, and document layout analysis. To our knowledge, this is the first work demonstrating that VLMs can generate faithful annotated document datasets at scale from unlabeled seeds that can effectively enrich or approximate real, manually annotated data for diverse document understanding tasks. We show that with only 100 real training samples, our framework achieves on average $87\%$ of the performance of the full real-world dataset. We publicly release our code and 140k+ synthetic document samples.

文档生成VLM合成数据扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。