arXiv:2511.20785cs.CV2025-11被引 56

让大模型像人一样看长视频,先概览后精读,减少幻觉。

LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling

  • 用模型自身时序定位能力当工具,自动剪裁关键片段逐层分析。
  • 在4个长视频基准上超越现有方法,准确率显著提升。
  • 公开15.4万样本数据集和代码,适合做长视频理解研究者使用。

大型多模态模型在视频推理中展现出巨大潜力,但处理长视频时易产生幻觉,因证据稀疏且时空分散。受人类观看长视频方式启发——先全局浏览再聚焦细节——我们提出LongVT,一种端到端的智能体框架,通过交错的多模态链式工具思维实现“思考长视频”。具体而言,利用大模型固有的时序定位能力作为原生视频裁剪工具,精准提取特定片段并重采样更细粒度帧。此全局到局部的推理循环持续进行,直至答案基于检索到的视觉证据。针对长视频问答数据稀缺问题,我们构建并发布名为VideoSIAH的数据套件:训练集包含247.9K样本用于工具集成冷启动微调、1.6K样本用于智能体强化学习、15.4K样本用于智能体强化微调;评估基准含1,280组经半自动流程与人工验证精心筛选的问答对。通过三阶段训练策略与充分实证验证,LongVT在四个挑战性长视频理解与推理基准上持续优于现有强基线。代码、数据与模型检查点已公开于https://github.com/EvolvingLMMs-Lab/LongVT。

原文摘要 · Abstract (English)

Large multimodal models (LMMs) have shown great potential for video reasoning with textual Chain-of-Thought. However, they remain vulnerable to hallucinations, especially when processing long-form videos where evidence is sparse and temporally dispersed. Inspired by how humans comprehend long videos - by first skimming globally and then examining relevant clips for details - we introduce LongVT, an end-to-end agentic framework that enables "Thinking with Long Videos" via interleaved Multimodal Chain-of-Tool-Thought. Specifically, we exploit LMMs' inherent temporal grounding ability as a native video cropping tool to zoom in on a specific video clip and resample finer-grained video frames. This global-to-local reasoning loop continues until answers are grounded in retrieved visual evidence. Given the scarcity of fine-grained question-answering (QA) data for the long video reasoning task, we curate and will release a data suite named VideoSIAH to facilitate both training and evaluation. Specifically, our training dataset consists of 247.9K samples for tool-integrated cold-start supervised fine-tuning, 1.6K samples for agentic reinforcement learning, and 15.4K samples for agentic reinforcement fine-tuning, respectively. Our evaluation benchmark consists of 1,280 QA pairs that are carefully curated through a semi-automatic data pipeline with human-in-the-loop validation. With a meticulously designed three-stage training strategy and extensive empirical validation, LongVT consistently outperforms existing strong baselines across four challenging long-video understanding and reasoning benchmarks. Our codes, data, and model checkpoints are publicly available at https://github.com/EvolvingLMMs-Lab/LongVT .

长视频理解多模态推理智能体视频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。