让视频研究模型自动选工具、会反思,减少错误和延迟。
AdaVDR: Adaptive Tool Use and Reflection for Video Deep Research

- 根据任务动态选择工具,只在结果不可靠时回溯修正。
- 在VDR-EE基准上超越现有开源模型,显著提升准确率。
- 适合需要精准视频理解与外部知识融合的研究场景。
视频深度研究通过联合理解视频内容与从开放网络检索外部知识来回答复杂问题。然而,多样的问题与视频需要不同的工具使用策略,不当的工具调用会导致错误结果。不确定的定位与检索使不必要的交互成本高且易出错,增加延迟和推理错误。为此,我们提出AdaVDR,一种具备自适应工具调用与反思能力的视频深度研究代理。AdaVDR根据任务及其能力选择工具,仅在中间结果不可靠时才回溯修正。为实现此能力,我们构建了视频深度研究数据构造流程:首先在多样视频中发现与检索相关事件和实体,通过定位与外部检索获取详细信息,构建高质量问答对。每个问答对使用任务特定提示将信息获取过程组织为工具使用轨迹,使不同问题与视频类型可采用不同的定位与检索策略。我们进一步引入模型条件化工具必要性过滤,基于目标模型的视频理解与内部知识评估工具调用,移除模型可跳过的工具或工具链,生成适配目标模型能力的轨迹。利用该流程,我们构建训练数据与VDR-EE基准,涵盖实体中心与事件中心问题。通过监督微调并结合冗余感知奖励进行强化学习,强化自适应工具调用与反思能力。实验表明,本方法在VDR-EE上表现最佳,显著优于其基线模型在VideoDR上的性能。
原文摘要 · Abstract (English)
Video deep research answers complex questions by jointly understanding video content and retrieving external knowledge from the open Web. However, diverse questions and videos require different tool-use strategies, and inappropriate tool calls can produce incorrect results. Uncertain grounding and retrieval also make unnecessary interactions costly and error-prone, increasing latency and reasoning errors. To address these challenges, we propose AdaVDR, an adaptive video deep research agent with adaptive tool invocation and reflection. AdaVDR selects tools according to the task and its capabilities, and backtracks only when unreliable intermediate results require correction. To enable these capabilities, we develop a video deep research data construction pipeline. We first discover retrieval-relevant events and entities in diverse videos and acquire detailed information through grounding and external retrieval to construct high-quality QA pairs. For each QA, task-specific prompts organize the information acquisition process into a tool-use trajectory, allowing different question and video types to follow different grounding and retrieval strategies. We further introduce model-conditioned tool necessity filtering, which evaluates tool calls against the target model's video understanding and internal knowledge, removing tools or tool chains the model can bypass. This yields trajectories tailored to the target model's video understanding capability and knowledge. Using this pipeline, we construct training data and VDR-EE, a benchmark covering entity-centric and event-centric questions. We perform supervised fine-tuning followed by reinforcement learning with a redundancy-aware reward to strengthen adaptive tool invocation and reflection. Experiments show that our method performs best among the evaluated open-source models on VDR-EE and substantially improves over its base models on VideoDR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。