DeepEyesV2让多模态模型能主动调用工具,像人类一样思考和行动。
DeepEyesV2: Toward Agentic Multimodal Model
- 分两阶段训练:先建立工具使用模式,再用强化学习优化调用策略。
- 在真实场景任务中表现优异,尤其擅长图像感知和数学推理。
- 支持根据任务自动选择工具,适合开发智能助手、自动决策系统。
代理式多模态模型不仅需理解文本与图像,还应主动调用外部工具(如代码执行环境、网络搜索),并将这些操作融入推理过程。本文提出 DeepEyesV2,从数据构建、训练方法到评估体系探索代理式多模态模型的构建路径。发现仅靠强化学习无法稳定诱导工具使用行为,因此设计两阶段训练流程:冷启动阶段建立工具使用模式,强化学习阶段进一步优化调用策略。构建了一个多样化且适中难度的训练数据集,包含工具使用带来显著收益的样本。同时推出 RealX-Bench 基准,用于评估真实世界多模态推理能力,涵盖感知、搜索与推理等多种能力集成。在 RealX-Bench 及其他基准上验证表明,DeepEyesV2 在真实理解、数学推理和搜索密集型任务中均表现优异。此外,其具备任务自适应工具调用能力,对感知任务倾向使用图像操作,对推理任务偏好数值计算。强化学习进一步支持复杂工具组合,并实现基于上下文的精准调用。本研究为社区构建代理式多模态模型提供实践指导。
原文摘要 · Abstract (English)
Agentic multimodal models should not only comprehend text and images, but also actively invoke external tools, such as code execution environments and web search, and integrate these operations into reasoning. In this work, we introduce DeepEyesV2 and explore how to build an agentic multimodal model from the perspectives of data construction, training methods, and model evaluation. We observe that direct reinforcement learning alone fails to induce robust tool-use behavior. This phenomenon motivates a two-stage training pipeline: a cold-start stage to establish tool-use patterns, and reinforcement learning stage to further refine tool invocation. We curate a diverse, moderately challenging training dataset, specifically including examples where tool use is beneficial. We further introduce RealX-Bench, a comprehensive benchmark designed to evaluate real-world multimodal reasoning, which inherently requires the integration of multiple capabilities, including perception, search, and reasoning. We evaluate DeepEyesV2 on RealX-Bench and other representative benchmarks, demonstrating its effectiveness across real-world understanding, mathematical reasoning, and search-intensive tasks. Moreover, DeepEyesV2 exhibits task-adaptive tool invocation, tending to use image operations for perception tasks and numerical computations for reasoning tasks. Reinforcement learning further enables complex tool combinations and allows model to selectively invoke tools based on context. We hope our study can provide guidance for community in developing agentic multimodal models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。