根据查询类型动态选择视频帧，提升长视频理解效率与准确率
Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understanding
- 按查询类型区分：全局查询用均匀采样，局部查询用相关帧提取
- 在三个基准上优于现有方法，256帧输入下仍保持高效
- 无需训练，适配大模型长视频任务，节省计算资源
大型多模态模型（LMMs）在长视频理解中受限于上下文长度和密集视频标记的高计算成本。现有研究聚焦于查询感知帧选择，但常伴随显著计算开销。本文提出挑战：复杂搜索机制并非普遍必需。通过识别并验证全局查询与局部查询的差异，发现均匀采样对全局查询既有效又高效，而局部查询需依赖查询感知选择以达最优性能。基于此，提出DIG框架——一种无需训练的帧选择方法，依据查询类型自适应策略：全局查询采用高效均匀采样，局部查询则激活专用管道提取相关帧。在三个长视频理解基准上的实验表明，DIG持续超越现有基线，即使输入帧数增至256时仍显著提升LMM表现。
原文摘要 · Abstract (English)
The application of Large Multimodal Models (LMMs) to long-form video understanding is constrained by limited context lengths and the computationally prohibitive cost of processing dense video tokens. Consequently, recent research has focused on query-aware frame selection, methods that often incur significant computational overhead. This paper challenges the assumption that such complex search mechanisms are universally necessary. We first identify and validate a query typology distinguishing between global query and localized query. We demonstrate that while uniform sampling is both effective and efficient for global queries, localized queries indeed necessitate query-aware selection for optimal performance. Building on this insight, we propose DIG, a training-free frame selection framework that adapts its strategy based on the query type. Specifically,DIG employs efficient uniform sampling for global queries while activating a specialized pipeline to extract query-relevant frames for localized queries. Experiments on three long-form video understanding benchmarks demonstrate that DIG consistently outperforms existing baselines and robustly improves LMM performance, even when scaling the input frame count to 256.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。