KTV通过选关键帧和关键视觉令牌,让视频大模型高效理解长视频。
KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs
- 先聚类选关键帧,再筛选每帧的重要视觉令牌,减少冗余
- 60分钟视频仅用504个视觉令牌,准确率达44.8%(MLVU-Test)
- 无需训练,比不少带训练的方法表现更好,适合资源受限场景
免训练视频理解利用预训练视觉语言模型(VLMs)强大的图像理解能力,将视频视为一系列静态帧序列,避免了昂贵的视频专用训练。然而,该范式常因视觉冗余和高计算开销而受困,尤其在处理长视频时更为严重。现有基于CLIP相似性的关键帧选择策略易产生偏差,可能遗漏关键帧,导致视频理解效果不佳。为此,我们提出新型两阶段框架KTV,实现高效且有效的免训练视频理解。第一阶段,通过聚类帧级视觉特征进行无问题依赖的关键帧选择,生成紧凑、多样且具有代表性的帧子集,缓解时间冗余;第二阶段,基于令牌重要性和冗余度,对每个选定的关键帧进行关键视觉令牌剪枝,显著减少输入到语言模型的令牌数量。在多选题视频问答任务上的大量实验表明,KTV优于当前最先进的免训练基线,同时使用极少的视觉令牌——例如,对于10800帧的60分钟视频仅需504个视觉令牌,在MLVU-Test基准上达到44.8%的准确率。特别地,KTV在某些基准上甚至超过多个训练型方法。
原文摘要 · Abstract (English)
Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision language models (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。