arXiv:2409.19573cs.CVcs.AI2024-09被引 4

用视觉定位增强关键信息提取,让模型先看后说。

See then Tell: Enhancing Key Information Extraction with Vision Grounding

  • 引入<see>标记定位图像区域,实现图文联动回答
  • 在CORD/SROIE/DocVQA上达到最新最佳性能
  • 构建含视觉定位的TableQA数据集,适合文档理解研究者

数字时代,理解融合文本、复杂排版与图像的视觉文档至关重要。传统关键信息提取(KIE)依赖光学字符识别(OCR),常带来延迟、计算开销和错误。现有先进图像到文本方法虽跳过OCR,但输出为无视觉定位的纯文本。本文提出STNet(See then Tell Net)端到端模型,实现精准答案与对应视觉定位。该模型使用独特<see>标记观察相关图像区域,解码器通过物理坐标解析此标记。答案开头的<see>标记使模型先“看”问题相关的图像区域,再“说”出结构化回答。为提升视觉感知能力,我们收集了大规模结构化表格识别数据集,并利用GPT-4构建包含精确视觉定位的TVG(TableQA with Vision Grounding)数据集。实验表明,本方法在CORD、SROIE和DocVQA等公开数据集上均达到领先性能,代码将开源。

原文摘要 · Abstract (English)

In the digital era, the ability to understand visually rich documents that integrate text, complex layouts, and imagery is critical. Traditional Key Information Extraction (KIE) methods primarily rely on Optical Character Recognition (OCR), which often introduces significant latency, computational overhead, and errors. Current advanced image-to-text approaches, which bypass OCR, typically yield plain text outputs without corresponding vision grounding. In this paper, we introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding. Distinctively, STNet utilizes a unique <see> token to observe pertinent image areas, aided by a decoder that interprets physical coordinates linked to this token. Positioned at the outset of the answer text, the <see> token allows the model to first see-observing the regions of the image related to the input question-and then tell-providing articulated textual responses. To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets. Leveraging the advanced text processing prowess of GPT-4, we develop the TVG (TableQA with Vision Grounding) dataset, which not only provides text-based Question Answering (QA) pairs but also incorporates precise vision grounding for these pairs. Our approach demonstrates substantial advancements in KIE performance, achieving state-of-the-art results on publicly available datasets such as CORD, SROIE, and DocVQA. The code will also be made publicly available.

关键信息提取视觉定位文档理解端到端模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。