让模型逐帧思考，精准挑选关键画面提升长视频理解能力
Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
- 用模型自己迭代筛选最相关的视频帧作为输入
- 在4个数据集上达到当前最佳，长视频表现尤其突出
- 适合需要精细长视频分析的研究与应用
尽管视觉语言模型（VLMs）取得进展，长视频理解仍是难题。现有先进模型虽可处理约1000帧，但仍难以有效利用长序列，易受上下文干扰。本文提出时间链式思维（Temporal Chain of Thought），一种视频问答的推理策略，通过模型自身迭代识别并提取最相关帧用于回答。实验表明，推理时增加计算量以精选上下文能显著提升准确率，符合近期大模型推理扩展的趋势。在4个多样化视频问答数据集上，本方法对3种不同VLM均实现一致性能提升。特别地，在超过1小时的长视频上，使用32K上下文窗口的方法，优于标准推理下70万帧窗口的相同VLM，性能高出2.8点。
原文摘要 · Abstract (English)
Despite recent advances in Vision-Language Models (VLMs), long-video understanding remains a challenging problem. Although state-of-the-art long-context VLMs can process around 1000 input frames, they still struggle to effectively leverage this sequence length, and succumb to irrelevant distractors within the context window. We present Temporal Chain of Thought, an inference strategy for video question-answering that curates the model's input context. We use the VLM itself to iteratively identify and extract the most relevant frames from the video, which are then used for answering. We demonstrate how leveraging more computation at inference-time to select the most relevant context leads to improvements in accuracy, in agreement with recent work on inference-time scaling of LLMs. Moreover, we achieve state-of-the-art results on 4 diverse video question-answering datasets, showing consistent improvements with 3 different VLMs. In particular, our method shines on longer videos which would not otherwise fit within the model's context window: On longer videos of more than 1 hour on LVBench, our approach using a context window of 32K outperforms the same VLM using standard inference with a 700K context window by 2.8 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。