针对长视频理解,提出自适应帧选择方法,兼顾相关性与覆盖性。
Adaptive Greedy Frame Selection for Long Video Understanding
- 基于问题自适应的贪心选帧,联合优化相关性与语义代表性。
- 在MLVU数据集上,不同帧数预算下均优于均匀采样和现有基线。
- 支持紧预算场景,适合需要高效处理长视频的任务。
大视觉-语言模型在长视频问答中应用日益广泛,但推理常受限于输入帧数和视觉标记数量。简单稀疏采样可能遗漏关键时刻,而纯相关性驱动的选择往往集中在近似重复帧上,牺牲了时间上远距离证据的覆盖。本文提出一种问题自适应的贪心帧选择方法,在固定帧预算下同时优化查询相关性和语义代表性。该方法构建1~FPS候选帧池(上限1000帧),精确对齐时间戳,使用SigLIP(相关性)和DINOv2(语义相似性)双空间嵌入候选帧,通过贪心策略最大化加权和:模块化相关性项与设施选址覆盖项。该目标函数满足归一化、单调性和子模性,可保证(1-1/e)近似率。为应对不同问题的相关性与覆盖性权衡,引入四种预设策略及轻量级文本仅问题类型分类器,动态路由至最优策略。在MLVU上的实验表明,该方法在各帧预算下均一致优于均匀采样和近期强基线,尤其在紧凑预算下提升显著。
原文摘要 · Abstract (English)
Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。