让多模态大模型像人一样深度检索,解决复杂视觉问题。
Vision-DeepResearch: Incentivizing DeepResearch Capability in Multimodal Large Language Models
- 通过多轮、多实体、多尺度搜索,突破传统单次查询局限。
- 支持数十轮推理与数百次引擎交互,显著提升答案准确性。
- 适合需要深度信息整合的复杂视觉问答场景。
多模态大语言模型(MLLMs)在众多视觉任务中表现卓越,但受限于内部世界知识容量,现有方法常采用‘推理-调用工具’策略,通过视觉与文本搜索引擎获取事实信息以提升性能。然而,这些方法通常假设一次全图或实体级图像查询及少量文本查询即可获取关键证据,这在存在大量视觉噪声的真实场景中不现实。同时,其推理深度与搜索广度有限,难以解决需整合多元视觉与文本来源证据的复杂问题。为此,我们提出 Vision-DeepResearch,一种新的多模态深度研究范式:通过多轮、多实体、多尺度的视觉与文本搜索,在高噪声环境下稳健调用真实搜索引擎。该方法支持数十轮推理与数百次引擎交互,并通过冷启动监督与强化学习将深度研究能力内化至 MLLM,形成端到端的强健多模态深度研究模型。实验表明,其显著优于现有 MLLMs 及基于 GPT-5、Gemini-2.5-pro、Claude-4-Sonnet 等闭源基座模型构建的工作。代码将开源于 https://github.com/Osilly/Vision-DeepResearch。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved remarkable success across a broad range of vision tasks. However, constrained by the capacity of their internal world knowledge, prior work has proposed augmenting MLLMs by ``reasoning-then-tool-call'' for visual and textual search engines to obtain substantial gains on tasks requiring extensive factual information. However, these approaches typically define multimodal search in a naive setting, assuming that a single full-level or entity-level image query and few text query suffices to retrieve the key evidence needed to answer the question, which is unrealistic in real-world scenarios with substantial visual noise. Moreover, they are often limited in the reasoning depth and search breadth, making it difficult to solve complex questions that require aggregating evidence from diverse visual and textual sources. Building on this, we propose Vision-DeepResearch, which proposes one new multimodal deep-research paradigm, i.e., performs multi-turn, multi-entity and multi-scale visual and textual search to robustly hit real-world search engines under heavy noise. Our Vision-DeepResearch supports dozens of reasoning steps and hundreds of engine interactions, while internalizing deep-research capabilities into the MLLM via cold-start supervision and RL training, resulting in a strong end-to-end multimodal deep-research MLLM. It substantially outperforming existing multimodal deep-research MLLMs, and workflows built on strong closed-source foundation model such as GPT-5, Gemini-2.5-pro and Claude-4-Sonnet. The code will be released in https://github.com/Osilly/Vision-DeepResearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。