通过分步聚焦关键布局区域,提升文档理解的精准度。
Doc-CoB: Enhancing Document Understanding with Visual Chain-of-Boxes Reasoning
- 采用从粗到细的视觉推理框架,逐步聚焦查询相关区域。
- 在7个基准上提升性能,最高显著优于基线模型。
- 适合需要精确文档信息提取的研究与应用者。
文档理解旨在对文档图像进行问答和信息抽取,其视觉内容信息密集,多数查询仅依赖少数相关版面区域。然而,现有方法或采用一通式策略,隐含假设所有版面同等重要;或过度关注小区域而丢失关键布局信息。为此,本文提出Doc-CoB(Chain-of-Boxes)框架,将粗粒度至细粒度的布局感知视觉推理融入多模态大语言模型。不同于直接放大小区域,Doc-CoB逐步聚焦于查询相关的版面区域,同时保留全局文档信息。具体而言,先选择关键版面框,再通过视觉提示深入理解。为支持该范式,引入两个推理任务:框识别与框推理,并构建包含24.9万样本的自动训练数据集,带有中间视觉监督。在7个基准、4种主流模型上的实验表明,Doc-CoB显著提升性能,验证其有效性与广泛适用性。代码与数据已公开于https://github.com/Doc-CoB/Doc-CoB。
原文摘要 · Abstract (English)
Document understanding aims to perform question answering and information extraction over document images, where the visual content is highly information-dense and most queries rely on only a few relevant layout regions. However, existing methods either adopt a one-pass strategy that implicitly assumes all layouts are equally important, or focus excessively on small regions at the cost of losing critical layout information. To address these limitations, we introduce Doc-CoB (Chain-of-Boxes), a simple-yet-effective framework that integrates coarse-to-fine layout-aware visual reasoning into multimodal large language models. Instead of directly zooming into small regions, Doc-CoB progressively focuses on query-relevant layouts while preserving global document information. Specifically, it first selects key layout boxes and then focuses on them for further understanding with visual prompting. To support this paradigm, we introduce two reasoning tasks for box recognition and box reasoning, with an automatic pipeline that constructs 249k training samples with intermediate visual supervision. Experiments on seven benchmarks with four popular models show that Doc-CoB significantly improves performance, demonstrating its effectiveness and wide applicability. The code and the data are available at https://github.com/Doc-CoB/Doc-CoB.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。