让视觉问答模型的思考过程可回溯,提升长文档答题可信度
AWM: Answerable Working Memory for Long-Document VQA Agents

- 设计记忆可回答性诊断,检验工作记忆是否包含完整答案证据
- 在长文档问答中,42.5%正确答案无法仅凭记忆回答,说明记忆质量差
- 新方法使模型既答对又保留可验证的推理痕迹,适合需要可解释性的场景
长文档视觉问答依赖于检索候选页面、分析图像、将发现写入工作记忆并合成答案的VLM代理。工作记忆需在多轮页面检查中持续携带支持答案的证据,但现有评估仅关注最终答案正确性和证据页访问情况,忽视了记忆质量。本文提出“仅凭记忆可回答性”诊断,判断读者能否仅凭问题与终态工作记忆作答。基于此,引入“可回答工作记忆”(AWM),将终态工作记忆视为可回答的证据载体,并将其信号融入GRPO奖励机制,同时保持最终答案优先。在 extsc{MMLongBench-Doc} 上,即使提供真实证据页,仍有42.5%的正确答案无法仅从终端工作记忆回答。AWM-GRPO在该数据集上比RAG基线提升8.1和11.9分,在 extsc{LongDocURL} 上也表现更优,且相比仅以答案为导向的GRPO,记忆缺失但答案正确的比例下降2.7个百分点。
原文摘要 · Abstract (English)
Long-document visual question answering increasingly relies on VLM agents that retrieve candidate pages, inspect page images, write findings to working memory, and synthesize answers. Working memory should carry answer-supporting evidence across page inspections for later grounded answering, yet existing evaluation mainly checks final-answer correctness and evidence-page access. This creates a memory-quality blind spot: an agent may reach the right page and answer correctly while leaving behind memory too generic or incomplete to support answering once page context is removed. We introduce \emph{memory-only answerability}, a diagnostic that asks whether a reader can answer from the question and terminal working memory alone. Building on this diagnostic, \emph{Answerable Working Memory} (AWM) treats terminal working memory as an answerable evidence artifact, and AWM-GRPO incorporates this signal into the GRPO reward while preserving final-answer priority. Under GRPO, this reward assigns higher advantages to answer-correct trajectories whose terminal working memory remains answerable. On \textsc{MMLongBench-Doc}, even when gold evidence pages are provided, 42.5\% of correct answers still cannot be answered from terminal working memory alone. AWM-GRPO improves final-answer accuracy over the RAG baseline by 8.1 and 11.9 points on \textsc{MMLongBench-Doc} and \textsc{LongDocURL} and reduces the memory-missing-correct rate by 2.7 points over answer-only GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。