用数学方法精准选出视频关键帧,用不到2%的帧实现长视频理解突破。
FOCUS: Efficient Keyframe Selection for Long Video Understanding
- 将关键帧选择建模为多臂赌博机问题,动态识别高信息量片段。
- 在长视频问答任务中仅用不到2%帧,准确率提升11.9%。
- 无需训练、兼容任意模型,适合大规模视频理解场景。
多模态大语言模型将图像和视频帧表示为视觉标记,但将单图扩展到时长一小时的视频会严重超出实际可承受的标记预算。现有方法通常采用均匀采样或依赖小型视觉-语言模型进行检索式打分的关键帧选择,但仍需预筛选以降低推理成本,可能遗漏关键信息。我们提出FOCUS(Frame-Optimistic Confidence Upper-bound Selection),一种无需训练、模型无关的关键帧选择模块,在严格标记预算下挑选与查询相关的帧。FOCUS将关键帧选择建模为多臂赌博机中的组合纯探索问题:将短时间段视为“臂”,利用经验均值与Bernstein置信半径识别信息丰富区域,同时保留对不确定区域的探索。由此产生的两阶段探索-利用流程具备理论保证,先定位高价值时间区域,再从中选取得分最高的帧。在两个长视频问答基准上,FOCUS在处理少于2%视频帧的情况下实现显著准确率提升;对于超过20分钟的视频,在LongVideoBench上达到11.9%的准确率增益,证明其作为关键帧选择方法的有效性,并为基于MLLM的可扩展长视频理解提供简单通用的解决方案。代码已公开于https://github.com/NUS-HPC-AI-Lab/FOCUS。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) represent images and video frames as visual tokens. Scaling from single images to hour-long videos, however, inflates the token budget far beyond practical limits. Popular pipelines therefore either uniformly subsample or apply keyframe selection with retrieval-style scoring using smaller vision-language models. However, these keyframe selection methods still rely on pre-filtering before selection to reduce the inference cost and can miss the most informative moments. We propose FOCUS, Frame-Optimistic Confidence Upper-bound Selection, a training-free, model-agnostic keyframe selection module that selects query-relevant frames under a strict token budget. FOCUS formulates keyframe selection as a combinatorial pure-exploration (CPE) problem in multi-armed bandits: it treats short temporal clips as arms, and uses empirical means and Bernstein confidence radius to identify informative regions while preserving exploration of uncertain areas. The resulting two-stage exploration-exploitation procedure reduces from a sequential policy with theoretical guarantees, first identifying high-value temporal regions, then selecting top-scoring frames within each region. On two long-video question-answering benchmarks, FOCUS delivers substantial accuracy improvements while processing less than 2% of video frames. For videos longer than 20 minutes, it achieves an 11.9% gain in accuracy on LongVideoBench, demonstrating its effectiveness as a keyframe selection method and providing a simple and general solution for scalable long-video understanding with MLLMs. Code is available at https://github.com/NUS-HPC-AI-Lab/FOCUS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。