让AI看视频并联网搜索,突破传统图文模型的局限。
Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

- 分阶段解锁工具,强制跨帧视觉定位后才上网查资料。
- 35B模型达64.0%准确率,超越Claude-4.5-Sonnet和GPT-5。
- 适合需要多步推理与视频理解的研究者与开发者。
我们提出Video-DeepResearch(Video-DR),将多模态智能体从静态图像扩展至连续视频流,该场景要求密集时空定位与开放式网络探索。初步评估揭示当前模型存在两大瓶颈:(1) 模态偏见,即智能体偏好文本搜索而绕过视觉工具;(2) 参数化知识泄露,模型依赖内部记忆而非真实工具增强执行。为此,我们设计了分离开感知与探索的流水线,采用分阶段工具解锁机制,强制在网页检索前完成跨帧视觉定位。框架采用两阶段训练策略:监督微调后接组相对策略优化(GRPO),实现突破模仿学习上限的自主探索。此外,我们构建了Video-DR-Bench,一个由人类与AI协作标注的基准,包含200个复杂、多跳的视频问答任务。实证结果表明,Video-DeepResearch-35B-A3B达到64.0%平均准确率,超越专有模型Claude-4.5-Sonnet(59.0%)5.0个百分点,并显著优于GPT-5(52.5%)和Gemini 2.5 Pro(57.5%)。30B-A3B版本达59.3%,在小规模下仍具竞争力,验证了训练范式的有效性。代码已开源。
原文摘要 · Abstract (English)
We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams, a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal memory rather than genuine tool-augmented execution. To address these challenges, we propose Video-DR, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval. Our framework adopts a two-stage training recipe: supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), enabling autonomous exploration that breaks the imitation-learning ceiling. Furthermore, we curate Video-DR-Bench, a human-AI collaborative benchmark comprising 200 complex, multi-hop VQA instances. Empirical results demonstrate that our Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 64.0% average accuracy, surpassing proprietary Claude-4.5-Sonnet (59.0%) by 5.0 points and significantly outperforming GPT-5 (52.5%) and Gemini 2.5 Pro (57.5%). The 30B-A3B variant achieves 59.3%, competitive with Claude-4.5-Sonnet and demonstrating the effectiveness of our training paradigm even at compact scale. Code: https://github.com/Osilly/Vision-DeepResearch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。