提出可渐进搜索的多模态代理，让视觉问答更智能地调用知识。
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
- 通过迭代选择图像/文本搜索或停止，动态调整检索策略。
- 在E-VQA和InfoSeek上提升检索与问答准确率，优于现有基线。
- 适合需要灵活知识调用的视觉问答场景，如复杂推理任务。
基于知识的视觉问答（KB-VQA）要求模型融合图像理解与外部知识。以往方法采用固定检索-生成流程，使用预设检索器和静态top-k设置，推理过程缺乏适应性。本文提出ProMSA，一种用于KB-VQA的渐进式多模态搜索代理。给定图像-问题对，该代理在显式工具调用预算下，迭代选择图像搜索、文本搜索或停止，并通过去重避免冗余检索。训练时，先用拒绝采样监督微调学习正确的工具调用格式，再通过TN-GSPO（一种序列级强化学习目标）优化，该目标同时按生成长度和工具交互深度归一化更新。在E-VQA和InfoSeek数据集上的实验表明，ProMSA持续优于强RAG与代理基线，显著提升检索与端到端准确率。代码已公开于https://github.com/DingWu1021/Promsa。
原文摘要 · Abstract (English)
Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。