让AI像人一样分段思考长视频，提升理解准确率
VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoning
- 用链式分段推理，逐步筛选与问题相关的视频片段
- 在VideoMME和EgoSchema上分别达到77.0和70.1的领先成绩
- 效率远超闭源模型，仅需7%帧数和12%推理时间
近期视频理解进展主要由多模态大模型（MLLMs）推动，但现有模型在处理长视频时表现不佳。为解决此问题，已有方法尝试使用MLLM作为智能体检索上下文，但普遍忽视长视频由多个镜头（shots）构成的本质。缺乏对相关镜头的深入理解，导致模型常误选冗余或噪声片段。为此，本文提出VideoChat-A1，一种新型长视频智能体范式，采用独特的链式分段推理机制：先逐步选取与用户问题相关的镜头，并以粗到细的方式深入分析；通过跨模态推理沿镜头链进行，模拟人类分步思考过程，实现对长视频中理想时间上下文的交互式发现。大量实验表明，VideoChat-A1在主流长视频问答基准上达到领先性能，如在VideoMME（含字幕）上达77.0，在EgoSchema上达70.1，相较InternVL2.5-8B和InternVideo2.5-8B分别提升最高10.1%和6.2%。相比GPT-4o和Gemini 1.5 Pro等闭源模型，其精度相当，但平均仅需7%输入帧和12%推理时间。代码已开源。
原文摘要 · Abstract (English)
Recent advances in video understanding have been driven by MLLMs. But these MLLMs are good at analyzing short videos, while suffering from difficulties in understanding videos with a longer context. To address this difficulty, several agent methods have been proposed, using MLLMs as agents for retrieving extra contextual knowledge in a long video. However, most existing agents ignore the key fact that a long video is composed with multiple shots, i.e., to answer the user question from a long video, it is critical to deeply understand its relevant shots like human. Without such insight, these agents often mistakenly find redundant even noisy temporal context, restricting their capacity for long video understanding. To fill this gap, we propose VideoChat-A1, a novel long video agent paradigm. Different from the previous works, our VideoChat-A1 can deeply think with long videos, via a distinct chain-of-shot reasoning paradigm. More specifically, it can progressively select the relevant shots of user question, and look into these shots in a coarse-to-fine partition. By multi-modal reasoning along the shot chain, VideoChat-A1 can effectively mimic step-by-step human thinking process, allowing the interactive discovery of preferable temporal context for thoughtful understanding in long videos. Extensive experiments show that, VideoChat-A1 achieves the state-of-the-art performance on the mainstream long video QA benchmarks, e.g., it achieves 77.0 on VideoMME (w/ subs) and 70.1 on EgoSchema, outperforming its strong baselines (e.g., InternVL2.5-8B and InternVideo2.5-8B), by up to 10.1\% and 6.2\%. Compared to leading closed-source GPT-4o and Gemini 1.5 Pro, VideoChat-A1 offers competitive accuracy, but only with 7% input frames and 12% inference time on average. The code is available on https://github.com/SpXace/VideoChat-A1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。