通过过程奖励训练多模态搜索智能体，实现零样本迁移并刷新性能纪录
ProMMSearchAgent: A Generalizable Multimodal Search Agent Trained with Process-Oriented Rewards
- 采用过程导向奖励机制，根据认知不确定性触发搜索
- 在沙盒环境中训练后，零样本迁移到真实谷歌搜索，性能提升超11%
- 适合需要高可靠性视觉推理的跨模态搜索应用
基于强化学习训练知识密集型多模态智能体面临结果反馈稀疏和实时网络环境不可预测的双重挑战。为此，我们提出ProMMSearchAgent，建立一种新型的仿真到现实训练范式。将策略学习解耦至确定性、局部静态沙盒环境。关键在于设计内省式过程导向奖励：通过探测代理自身的参数化知识边界，生成密集的行为元数据，仅在视觉或事实不确定时才触发多模态或文本搜索，从而显式奖励正确决策。大量实验表明，该本地训练策略可零样本迁移至真实谷歌搜索接口。ProMMSearchAgent在FVQA-test上优于MMSearch-R1 5.1%，InfoSeek上提升6.3%，MMSearch上提升11.3%，达到新SOTA。
原文摘要 · Abstract (English)
Training multimodal agents via reinforcement learning for knowledge-intensive visual reasoning is fundamentally hindered by the extreme sparsity of outcome-based supervision and the unpredictability of live web environments. To resolve these algorithmic and environmental bottlenecks, we introduce ProMMSearchAgent, establishing a novel Sim-to-Real training paradigm for multimodal search. We decouple policy learning into a deterministic, local static sandbox. Crucially, to learn effectively within this constrained environment, we propose an introspective process-oriented reward. By probing the agent's own parametric knowledge boundaries, we generate dense behavioral metadata that explicitly rewards the correct cognitive decision, initiating a multimodal or text search only when visually or factually uncertain. Extensive experiments demonstrate that our locally-trained policy transfers zero-shot to the live Google Search API. ProMMSearchAgent achieves new SOTA performance, outperforming MMSearch-R1 by +5.1% on FVQA-test, +6.3% on InfoSeek, and +11.3% on MMSearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。