arXiv:2604.07146cs.CV2026-04被引 2

让模型像人一样决策该不该查、查什么,提升视觉问答准确率。

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

论文配图:Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering
图 1 · 摘自论文原文
  • 把问答变成多步决策过程,动态选择查图、查文或直接答。
  • 在InfoSeek和E-VQA上达到最新最好效果,显著优于传统方法。
  • 适合需要灵活推理的复杂视觉问答任务,尤其对罕见实体有效。

基于知识的视觉问答(KB-VQA)要求视觉语言模型理解图像并利用外部知识,特别是针对罕见实体和长尾事实。现有检索增强生成(RAG)方法多采用固定流水线,依次检索、过滤并生成答案,难以适应多样问题类型。同时,检索与推理分离,导致模型难以决定何时搜索、如何优化查询或何时停止,致使检索证据与问题匹配度低。为此,本文将KB-VQA重构为搜索代理问题,将求解过程建模为多步决策过程。每一步中,代理基于当前信息状态选择四类动作之一:回答、图像检索、文本检索或基于标题的检索。我们进一步设计自动化流程,收集包含推理过程、工具使用和中间决策的多步轨迹,并以此作为微调监督信号。在InfoSeek和E-VQA上的实验表明,本方法达到当前最佳性能,持续超越已有基线,验证了框架的有效性。

原文摘要 · Abstract (English)

Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.

视觉问答决策代理知识增强多步推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。