arXiv:2511.12003cs.AI2025-11AAAI被引 5

让AI推理时像思考一样看图,精准定位答案依据。

Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning

  • 用强化学习训练模型边推理边标注图像区域和页码
  • 在双数据集上提升8.23%答案准确率,定位精度提升47%
  • 适合需要可验证答案的学术文档问答场景

为从视觉文档中精确定位证据来源,视觉文档检索增强生成(VD-RAG)需确保多模态问答中视觉语言模型(VLMs)预测的可靠性和可验证性。现有方法多采用端到端训练以实现直观的答案验证,但缺乏细粒度监督与推理过程的逐步追溯能力。本文提出链式证据(Chain-of-Evidence, CoE)范式,将思维链(CoT)推理与视觉证据归属统一,通过边界框和页码索引将推理步骤中的参考元素锚定至具体图像区域。为此,我们设计了‘看即所思’(Look As You Think, LAT)强化学习框架,使VLM能生成具有可验证性的推理路径。训练中,LAT评估每处证据区域的归属一致性,并仅当整个CoE轨迹得出正确答案时才给予奖励,推动过程级自验证。在Qwen2.5-VL-7B-Instruct模型上,结合Paper-VISA与Wiki-VISA基准测试表明,LAT在单图与多图设置下均显著优于基线模型,软精确匹配(soft EM)平均提升8.23%,[email protected]提升47.0%。同时,其性能超越直接输出带归属信息答案的监督微调基线,并展现出更强跨领域泛化能力。

原文摘要 · Abstract (English)

Aiming to identify precise evidence sources from visual documents, visual evidence attribution for visual document retrieval-augmented generation (VD-RAG) ensures reliable and verifiable predictions from vision-language models (VLMs) in multimodal question answering. Most existing methods adopt end-to-end training to facilitate intuitive answer verification. However, they lack fine-grained supervision and progressive traceability throughout the reasoning process. In this paper, we introduce the Chain-of-Evidence (CoE) paradigm for VD-RAG. CoE unifies Chain-of-Thought (CoT) reasoning and visual evidence attribution by grounding reference elements in reasoning steps to specific regions with bounding boxes and page indexes. To enable VLMs to generate such evidence-grounded reasoning, we propose Look As You Think (LAT), a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution. During training, LAT evaluates the attribution consistency of each evidence region and provides rewards only when the CoE trajectory yields correct answers, encouraging process-level self-verification. Experiments on vanilla Qwen2.5-VL-7B-Instruct with Paper- and Wiki-VISA benchmarks show that LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in [email protected]. Meanwhile, LAT not only outperforms the supervised fine-tuning baseline, which is trained to directly produce answers with attribution, but also exhibits stronger generalization across domains.

视觉推理可验证生成强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。