用强化学习让多模态模型自己验证检索信息,减少对外部验证依赖。
Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning

- 通过强化学习训练模型在推理中自我验证和筛选检索内容。
- 在5000个视觉问答样本上微调后,多跳问答泛化能力显著提升。
- 适合追求高效、低成本且高可靠性的多模态推理系统开发者。
多模态推理模型日益依赖外部工具(如网络搜索)回答复杂问题。尽管强化学习(RL)微调算法(如GRPO)提升了纯文本语言模型在编程与数学任务中的长序列推理能力,但多模态模型在工具使用上的可靠性仍面临挑战:模型需同时理解文本与图像,并整合噪声较大的检索证据,通常在缺乏明确验证信号的稀疏结果级监督下进行。本文提出基于强化学习的自验证框架SVRL,仅通过强化学习训练多模态模型在自身推理过程中验证并过滤检索证据,从而在推理阶段无需依赖外部验证器。SVRL引入搜索感知惩罚项,抑制不必要的工具调用;设计查询多样性奖励,鼓励生成多样且结构良好的搜索查询,提供细粒度反馈以指导何时以及如何搜索。在仅5,000个视觉问答样本上对Qwen-2.5-VL-7B进行微调,即可在多个基准测试中实现多跳VQA泛化性能提升与工具使用效率优化。总体而言,SVRL显著缩小了轻量级模型与大型专有模型之间的差距,同时大幅降低训练与推理成本。
原文摘要 · Abstract (English)
Reasoning agents increasingly rely on external tools such as web search to answer complex queries. Reinforcement learning (RL) finetuning algorithms such as GRPO have improved long-form reasoning in text-only language models, particularly for coding and mathematics. Reliable tool use in multimodal agents, however, remains challenging because models must interpret text and images while integrating noisy retrieved evidence, often under sparse outcome-level supervision without explicit verification signals. We present Self-Verification via Reinforcement Learning (SVRL), an RL-only finetuning framework that trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces, reducing reliance on external verifiers at inference time. SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries, providing fine-grained feedback on when and what to search. Finetuning Qwen-2.5-VL-7B with SVRL on only 5{,}000 visual question answering examples yields consistent gains in multi-hop VQA generalization and tool efficiency across benchmarks. Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。