让视频大模型学会根据问题主动挑选关键帧，提升理解效率。
Frame-Voyager: Learning to Query Frames for Video Large Language Models
- 基于任务文本动态查询高信息量的帧组合
- 在4个视频问答基准上均显著优于传统采样方法
- 可直接接入现有视频大模型，适合需要高效处理长视频的场景
视频大语言模型在视频理解任务中取得了显著进展，但受限于输入令牌长度上限，难以处理完整视频。现有帧选择方法如均匀采样和文本-帧检索，未考虑视频中信息密度差异及任务指令复杂性，导致性能不佳。本文提出 Frame-Voyager，根据任务中的文本查询学习选择信息丰富的帧组合。为训练该模型，我们设计了一种新的数据收集与标注流程：对一个含 M 帧的视频，遍历其所有 T 帧组合，输入预训练视频大模型并根据预测损失进行排序，以此作为监督信号训练 Frame-Voyager 学会查询低损失的帧组合。在两个不同视频大模型上插入 Frame-Voyager，并在四个视频问答基准上测试，结果表明其在所有设置下均表现优异，展现出作为即插即用解决方案的巨大潜力。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。