arXiv:2512.00805cs.CV2025-12中稿 · CVPR被引 2

用草稿机制提升长视频理解效率,推理速度更快且准确

Thinking with Drafts: Speculative Temporal Reasoning for Efficient Long Video Understanding

  • 用轻量草稿模型预选关键帧,主模型专注推理验证
  • 在多个数据集上推理速度提升显著,精度仍保持领先
  • 适合需要高效处理长视频的多模态大模型应用

长视频理解对实现类人智能至关重要,能支持对长时间上下文的连贯感知与推理。尽管新兴的‘思考-帧’范式(交替进行全局时序推理与局部帧分析)提升了多模态大语言模型(MLLM)的推理能力,但其面临因逐步增长且冗余的多模态上下文导致的严重效率瓶颈。为此,我们提出SpecTemp:一种基于强化学习的推测性时序推理框架,通过协作双模型设计将时序感知与推理解耦。在SpecTemp中,轻量级草稿MLLM快速探索并提出密集采样时序区域中的显著帧,而强大的目标MLLM则专注于时序推理并验证草稿提议,迭代精炼注意力直至收敛。该设计模拟人类大脑的协同路径,在效率与准确性间取得平衡。为支持训练,我们构建了SpecTemp-80K数据集,包含粗粒度证据段与细粒度帧级证据的同步双层标注。在多个视频理解基准上的实验表明,SpecTemp不仅保持了竞争力的精度,还显著加速了推理过程,优于现有‘思考-帧’方法。

原文摘要 · Abstract (English)

Long video understanding is essential for human-like intelligence, enabling coherent perception and reasoning over extended temporal contexts. While the emerging thinking-with-frames paradigm, which alternates between global temporal reasoning and local frame examination, has advanced the reasoning capabilities of video multi-modal large language models (MLLMs), it suffers from a significant efficiency bottleneck due to the progressively growing and redundant multi-modal context. To address this, we propose SpecTemp, a reinforcement learning-based Speculative Temporal reasoning framework that decouples temporal perception from reasoning via a cooperative dual-model design. In SpecTemp, a lightweight draft MLLM rapidly explores and proposes salient frames from densely sampled temporal regions, while a powerful target MLLM focuses on temporal reasoning and verifies the draft's proposals, iteratively refining its attention until convergence. This design mirrors the collaborative pathways of the human brain, balancing efficiency with accuracy. To support training, we construct the SpecTemp-80K dataset, featuring synchronized dual-level annotations for coarse evidence spans and fine-grained frame-level evidence. Experiments across multiple video understanding benchmarks demonstrate that SpecTemp not only maintains competitive accuracy but also significantly accelerates inference compared with existing thinking-with-frames methods.

视频理解推理加速多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。