让AI在看不懂图时会问问题，提升视觉问答准确率
Clarify or Answer: Reinforcement Learning for Agentic VQA with Context Under-specification
- 设计可选择提问或直接回答的智能体，分步决策是否需要澄清
- 在三个数据集上平均提升15.3分，端到端准确率提高83%
- 适合需要高可靠性的视觉理解场景，如医疗、司法辅助
现实世界的视觉问答常依赖上下文：图像与问题组合可能信息不足，正确答案需外部补充信息。此时直接作答易产生自信但错误的结果。我们提出CoA（Clarify-or-Answer），一种可区分提问或回答决策的智能体，先判断是否需要澄清；若需，则提出一个聚焦问题并结合回复生成最终答案。我们构建CONTEXTCLARIFY数据集，包含一组模糊问题及其对应的非模糊对照集。进一步提出GRPO-CR（Clarification Reasoning）强化学习方法，通过多重奖励信号优化提问质量，鼓励生成语法正确、聚焦且非平凡的问题以解决歧义。在三个视觉语言大模型和三个数据集上，CoA在模块与系统层面均取得一致提升，相比基于提示的基线，端到端VQA准确率平均提高15.3分（83%）。
原文摘要 · Abstract (English)
Real-world visual question answering (VQA) is often context-dependent: an image-question pair may be under-specified, such that the correct answer depends on external information that is not observable in the image. In such cases, directly answering can lead to confident but incorrect predictions. We propose CoA(Clarify-or-Answer), an ask-or-answer agent that separately models the decision to ask or answer, and what to ask if needed. CoA first determines whether clarification is necessary; if so, it asks a single focused question and then incorporates the response to produce the final answer. We introduce CONTEXTCLARIFY with a set of ambiguous VQA questions and the contrast set that is non-ambiguous. We further introduce GRPO-CR (Clarification Reasoning), a reinforcement learning approach that optimizes clarification question generation with multiple reward signals encouraging well-formed, focused, non-trivial questions that resolve ambiguity. Across three VLLMs and three datasets, CoA achieves consistent improvements at both the module and system levels, improving end-to-end VQA accuracy by an average of +15.3 points (83%) over prompting-based baselines
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。