用大模型生成含复杂关系的文档,提升视觉信息提取效果
Relation-Rich Visual Document Generator for Visual Information Extraction
- 分两阶段生成:先用大模型构造带关系的文本,再根据OCR结果生成多样布局
- 在多个VIE基准上显著提升下游模型性能,无需人工标注
- 适合需要高质量合成文档数据的研究者和开发者
尽管大型语言模型(LLMs)和多模态大模型(MLLMs)在视觉文档理解(VDU)方面取得进展,但富含关系的文档中视觉信息提取(VIE)仍面临版式多样性和训练数据不足的挑战。现有合成文档生成方法或依赖手工设计版式模板,或采用规则驱动方式,限制了版式多样性。此外,当前布局生成方法仅关注拓扑结构,未考虑文本内容,导致难以生成内容与布局紧密关联的复杂文档。本文提出一种关系丰富的视觉文档生成器(RIDGE),采用两阶段方法:(1) 内容生成阶段,利用大模型基于精心设计的分层结构文本格式生成包含实体类别与关系的文档内容;(2) 内容驱动的版式生成阶段,仅基于易获取的光学字符识别(OCR)结果学习生成多样化、合理的版式,无需人工标注或注释。实验表明,该方法显著提升了文档理解模型在多个VIE基准上的表现。代码与模型将公开于https://github.com/AI-Application-and-Integration-Lab/RIDGE。
原文摘要 · Abstract (English)
Despite advances in Large Language Models (LLMs) and Multimodal LLMs (MLLMs) for visual document understanding (VDU), visual information extraction (VIE) from relation-rich documents remains challenging due to the layout diversity and limited training data. While existing synthetic document generators attempt to address data scarcity, they either rely on manually designed layouts and templates, or adopt rule-based approaches that limit layout diversity. Besides, current layout generation methods focus solely on topological patterns without considering textual content, making them impractical for generating documents with complex associations between the contents and layouts. In this paper, we propose a Relation-rIch visual Document GEnerator (RIDGE) that addresses these limitations through a two-stage approach: (1) Content Generation, which leverages LLMs to generate document content using a carefully designed Hierarchical Structure Text format which captures entity categories and relationships, and (2) Content-driven Layout Generation, which learns to create diverse, plausible document layouts solely from easily available Optical Character Recognition (OCR) results, requiring no human labeling or annotations efforts. Experimental results have demonstrated that our method significantly enhances the performance of document understanding models on various VIE benchmarks. The code and model will be available at https://github.com/AI-Application-and-Integration-Lab/RIDGE .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。