自适应选择视频关键帧,大幅减少计算量同时保持精度。
Coverage-Driven Adaptive Keyframe Selection for Video Understanding

- 根据语义重要性动态决定评分与选帧数量,实现智能覆盖。
- 相比基线少评4~13倍帧,少选18.4%~20.5%关键帧。
- 无需训练,适用于多种视觉语言模型,适合长视频理解任务。
大型视觉语言模型(LVLMs)在长视频理解方面取得进展,但处理大量帧带来巨大计算开销。现有方法通过预评分帧-查询相关性并选取关键帧来降低推理成本。然而,相关帧分布随查询变化,常需评分数百至数千帧。为此,我们提出无需训练的语义关键帧选择器CSES,可自适应确定评分帧数与选帧数。CSES通过估计帧-查询相关性分布的重要性,引导主动获取并调整输入的时间覆盖范围,将关键帧选择建模为兼顾语义相关性、时间冗余与视觉冗余的覆盖问题。主动获取与选帧基于覆盖饱和终止。该目标函数单调且子模,支持贪婪优化并保证标准近似率。在两个基准上使用四种LVLM的实验表明,本方法在保持精度的同时,仅需评分4~13倍更少帧,选出18.4%~20.5%更少的输入关键帧,且帧选择速度比基线快3.1~5.4倍。
原文摘要 · Abstract (English)
Recent advances in large vision-language models (LVLMs) have enabled long-video understanding and analysis. However, processing the large number of frames in a video incurs substantial computational overhead. Existing methods reduce LVLM inference costs by scoring frame-query relevance before inference and selecting keyframes accordingly. Nevertheless, the distribution of relevant frames varies across queries, and these methods often need to score hundreds or thousands of frames. To address this limitation, we propose CSES, a training-free semantic keyframe selector that adaptively determines the numbers of frames to score and keyframes to select. CSES estimates the prominence of the frame-query relevance profile to guide active acquisition and adapt the temporal coverage of each input. It then formulates keyframe selection as a coverage problem that jointly accounts for semantic relevance, temporal redundancy, and visual redundancy. Active acquisition and keyframe selection terminate based on coverage saturation. The selection objective is monotone and submodular, enabling greedy optimization with a standard approximation guarantee. Experiments with four LVLMs on two benchmarks show that our method preserves accuracy while scoring $4$-$13\times$ fewer frames and selecting $18.4\%$-$20.5\%$ fewer input keyframes than existing baselines. CSES further achieves a $3.1$-$5.4\times$ speedup in frame selection over baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。