让模型按需检索文档图像细节,提升细粒度识别效率与精度。
State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models

- 根据当前解码状态动态检索关键图像区域,避免重复读取全图。
- 在低分辨率输入下仍保持高鲁棒性,准确率与效率双提升。
- 适合需要精细图文理解的文档分析任务,如发票、表单识别。
与典型视觉语言任务相比,文档解析对细粒度视觉感知要求更高。现有基于视觉语言模型(VLM)的解析方法依赖全局压缩的视觉标记,导致细粒度信息被混杂于单一表征中,并在解码过程中反复访问。然而我们发现,每个预测所需的视觉证据通常是局部且依赖当前解码状态的,而现有方法却在每一步都需完整访问这些表示,造成计算低效。为此,我们在自回归解码中提出状态条件视觉证据检索(SCVER),模型使用紧凑的全局表征捕捉粗粒度结构,并根据当前标记状态检索一组相关高分辨率区域。这种从粗到精的设计实现细粒度视觉线索的按需访问,减轻全局共享表征对所有细粒度细节的编码负担。我们进一步发现,在VLM中学习这种状态条件检索具有挑战性和不稳定性。为此,引入空间引导学习目标(SGLO)以稳定检索过程。在多个文档解析基准上的实验表明,SCVER在降低输入分辨率时仍具更强鲁棒性,并实现了更优的准确率-效率权衡,验证了按需视觉证据检索在细粒度感知中的有效性。
原文摘要 · Abstract (English)
Compared with typical vision-language tasks, document parsing places stronger demands on fine-grained visual perception. Existing vision-language model (VLM)-based parsing approaches rely on globally compressed visual tokens, where fine-grained details are entangled within a single representation and repeatedly accessed during decoding. However, we observe that the visual evidence for each prediction is typically localized and conditioned on the current decoding state, whereas such representations must be accessed in full at every decoding step, resulting in inefficient computation. To address this mismatch, we formulate perception as state-conditioned visual evidence retrieval (SCVER) during autoregressive decoding. The model operates on a compact global representation for coarse structure and retrieves a small set of relevant high-resolution regions conditioned on the current token state. This coarse-to-fine design enables on-demand access to fine-grained visual cues, relieving globally shared representations from encoding all fine-grained details. We further find that learning such state-conditioned retrieval in VLMs is challenging and unstable. To stabilize this process, we introduce a Spatially-Guided Learning Objective (SGLO) to guide the retrieval process. Experiments on document parsing benchmarks show that SCVER improves robustness under reduced input resolution and achieves a better accuracy-efficiency trade-off, demonstrating the effectiveness of on-demand visual evidence retrieval for fine-grained perception.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。