提升多模态搜索代理的深度推理能力，解决过早交互崩溃问题。
DR-MMSearchAgent: Deepening Reasoning in Multimodal Search Agents
- 基于结构相似性从全批次轨迹中提取优势信号，鼓励不同长度路径生成。
- 引入差异化高斯奖励，动态调节交互容忍度，减少冗余信息。
- 构建3602对三步以上推理的高质量数据集，适合复杂多轮任务研究者。
代理型多模态模型因其能调用外部工具处理复杂任务而备受关注。然而，这类代理常因两个原因出现过早交互崩溃：一是终端奖励仅附加在最后一个标记上，导致难以区分探索性行为的优劣；二是上下文过度冗余，阻碍代理吸收有效反馈。为此，我们提出深度推理多模态搜索代理（DR-MMSearchAgent），该框架利用结构相近性，在整个批次内从完整轨迹中推导优势信号，使不同长度的轨迹均被鼓励生成，即使包含相同正确答案。同时，采用差异化的高斯奖励动态校准交互容忍度，确保信息可靠性并降低冗余。为支持多轮交互训练，我们构建了一个包含3602个高质量问答对的多步深度推理数据集，每组至少包含3个推理步骤。大量实验表明，本方法在FVQA-test上较MMSearch-R1提升8.4%。
原文摘要 · Abstract (English)
Agentic multimodal models have garnered significant attention for their ability to leverage external tools to tackle complex tasks. However, it is observed that such agents often meet premature interaction collapse, caused by two primary reasons: 1) the terminal reward often appending on the last token prevents the advantage from distinguishing trajectories with exploratory behavior; 2) excessively redundant context hinders the agent from absorbing useful feedback. To address these issues, we propose the Deepening Reasoning MMSearchAgent, the framework leverages the structural proximity to derive advantage signals from the whole rollout trajectories in an entire batch, such that trajectories of different lengths are further encouraged to be generated, even when containing the same correct answer. Additionally, differentiated gaussian rewards are employed to dynamically calibrate interaction tolerance, thereby ensuring information reliability and reduce redundancy. To support multi-turn interaction training, we have constructed a multi-step deep-reasoning dataset including 3602 high-quality QA pair with at least 3 reasonning steps. Extensive experiments demonstrate that our method achieves state-of-the-art performance, outperforming the MMSearch-R1 by 8.4$\%$ on FVQA-test.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。