用合成工具轨迹训练视频大模型,实现动态探索长视频关键片段。
VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning
- 通过生成多步工具使用序列,在文本空间构建合成数据
- 在长视频基准上超越现有模型,显著提升时序定位能力
- 适合需要精准理解长视频的科研与工业用户
长视频理解仍是当前视频大语言模型的核心挑战。现有模型多依赖均匀采样帧进行静态推理,导致时序定位能力弱且信息损失严重。自适应工具如时间检索、空间缩放和时间缩放可有效突破此局限。但构建此类智能体数据需模型本身已具备强视频理解能力,形成循环依赖。我们提出 VideoThinker,一种完全基于合成工具交互轨迹训练的智能体视频大模型。核心思路是将视频转化为丰富描述,并利用强大语言模型生成多步工具使用序列;再将这些描述替换为对应帧,实现从文本到视频的对齐,构建大规模交错视频与工具推理数据集,无需底层模型具备长视频理解能力。在该合成数据上训练使 VideoThinker 获得动态推理、自适应时序探索及多步工具使用能力。显著优于仅依赖描述的语言模型代理和强视频模型基线,在多个长视频基准上表现优异,验证了工具增强合成数据与自适应检索/缩放推理的有效性。
原文摘要 · Abstract (English)
Long-form video understanding remains a fundamental challenge for current Video Large Language Models. Most existing models rely on static reasoning over uniformly sampled frames, which weakens temporal localization and leads to substantial information loss in long videos. Agentic tools such as temporal retrieval, spatial zoom, and temporal zoom offer a natural way to overcome these limitations by enabling adaptive exploration of key moments. However, constructing agentic video understanding data requires models that already possess strong long-form video comprehension, creating a circular dependency. We address this challenge with VideoThinker, an agentic Video Large Language Model trained entirely on synthetic tool interaction trajectories. Our key idea is to convert videos into rich captions and employ a powerful agentic language model to generate multi-step tool use sequences in caption space. These trajectories are subsequently grounded back to video by replacing captions with the corresponding frames, yielding a large-scale interleaved video and tool reasoning dataset without requiring any long-form understanding from the underlying model. Training on this synthetic agentic dataset equips VideoThinker with dynamic reasoning capabilities, adaptive temporal exploration, and multi-step tool use. Remarkably, VideoThinker significantly outperforms both caption-only language model agents and strong video model baselines across long-video benchmarks, demonstrating the effectiveness of tool augmented synthetic data and adaptive retrieval and zoom reasoning for long-form video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。