用智能代理分段搜索视频,让大模型更准理解长视频。
Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding
- 设计可自适应的智能体,按需调用工具处理不同问题。
- 在LVBench上达74.2%准确率,用字幕提升至76.0%。
- 适合需要精读长视频的科研与内容分析场景。
长时视频理解因时空复杂性高、上下文长而面临挑战。尽管大语言模型在视频分析和长上下文处理上取得进展,但在信息密集的小时级视频中仍存局限。为此,我们提出深度视频发现(Deep Video Discovery, DVD)智能体,通过分段视频片段的智能搜索策略克服限制。不同于以往统一预设流程的视频智能体,本方法强调智能体的自主与自适应能力。在多粒度视频数据库上提供以搜索为中心的一组工具,DVD智能体利用大模型的推理能力,基于当前观察状态规划行动,动态选择工具以针对不同问题构建适应性工作流。我们在多个长视频理解基准上进行综合评估,结果表明该方法具有显著优势:在挑战性数据集LVBench上达到74.2%准确率,显著超越此前所有方法;结合字幕后进一步提升至76.0%。代码已开源:https://github.com/microsoft/DeepVideoDiscovery。
原文摘要 · Abstract (English)
Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. To overcome such limitations, we propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips. Unlike previous video agents that rely on predefined workflows applied uniformly across different queries, our approach emphasizes the autonomous and adaptive nature of agents. By providing a set of search-centric tools on multi-granular video database, our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information. We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage. Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%, which substantially surpasses all prior works, and further improves to 76.0% with transcripts. The code has been released at https://github.com/microsoft/DeepVideoDiscovery.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。