一种可适配任意帧数预算的视频选帧方法,提升长视频理解效率。
One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding
- 构建层级式优先级序列,早期聚焦关键证据,后期逐步扩展时间上下文。
- 在4个基准上相比均匀采样平均提升3.77%准确率,支持6种预算灵活切换。
- 无需训练,通过稀疏探测和贪心排序实现高效选帧,降低50%以上延迟。
由于长视频存在严重帧冗余且大模型上下文窗口有限,帧选择对应用大型多模态模型至关重要。现有方法通常为每种固定预算单独优化选帧集,导致预算变化时需重新计算,无法复用。本文提出马特罗什卡证据-上下文(MEC)选帧框架,将长视频帧选择建模为马特罗什卡排名问题:构建单一优先级序列,其小前缀集中查询相关的关键证据,更大前缀在保留证据的同时逐步增加时间覆盖范围。为高效构建该序列,采用无需训练的稀疏视频索引,通过稀疏探测与局部放大发现候选帧,并贪心生成位置自适应的排序——早期强调证据,后期逐步偏向时间覆盖与视觉多样性。单个排序可截断至任意目标预算而无需重新运行选择器。在四个基准、六种帧预算下,MEC相较均匀采样平均提升3.77个百分点准确率,达到先进方法水平,端到端选帧延迟降低47.37%-51.19%。
原文摘要 · Abstract (English)
Frame selection is essential for applying Large Multimodal Models (LMMs) to long videos due to severe frame redundancy and limited context windows. Since the appropriate frame budget varies with the downstream LMM, reasoning demands, and latency constraints, a practical selector should serve multiple budgets. However, existing methods typically optimize an isolated frame subset for each predefined budget: when the budget changes, previously selected evidence may be replaced rather than progressively augmented. Ranking frames by a fixed score would allow prefix reuse across budgets, but it ignores the distinct roles of different ranking positions. In this paper, we formulate long-video frame selection as a Matryoshka ranking problem: constructing a single priority sequence whose small prefixes concentrate query-conditioned evidence, while progressively larger prefixes preserve this evidence and add broader temporal context. Efficiently constructing such a ranking is itself challenging, as densely sampling long videos and evaluating frame-query relevance incurs substantial overhead. We therefore introduce Matryoshka Evidence-to-Context (MEC) Frame Selection, a training-free framework that builds a reusable sparse video index, discovers candidates through sparse probing and local zooming, and greedily constructs a position-adaptive ranking: early positions emphasize evidence; later positions progressively favor temporal coverage while preserving visual diversity. A single ranking can thus be truncated to any target budget without rerunning the selector. Across four benchmarks and six frame budgets, MEC improves average accuracy over uniform sampling by 3.77 percentage points, matches strong state-of-the-art selectors, and reduces end-to-end selection latency by 47.37-51.19%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。