GIFT通过评估帧的不可替代性，高效选取关键视频帧，提升模型理解精度。
GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding
- 基于帧的不可替代性评分，统一衡量相关性与独特性。
- 在长视频上相较均匀采样提升12.5%平均性能。
- 无需训练，适合资源受限场景下的视频理解应用。
视频大语言模型在视频理解任务中表现优异，但密集帧处理带来的巨大计算开销严重限制了其实际应用。现有方法通过选择关键帧缓解该问题，但其贪婪决策方式与相关性、多样性解耦评估常陷入局部最优，导致错误选择无关噪声帧。为此，我们提出GIFT：全局不可替代性帧定位，一种无需训练的框架，通过评估帧的内在不可替代性来选择关键帧。首先引入有向多样性，量化在相关性条件下的帧独特性，从而构建统一的不可替代性评分；随后采用预算感知精炼策略，通过自适应迭代过程，先锁定高不可替代性的核心帧集，再随预算扩展逐步强化这些选帧的时序上下文。大量实验表明，相较于均匀采样，GIFT在长视频基准上对LLaVA-Video-7B实现最高12.5%的平均性能提升。
原文摘要 · Abstract (English)
Video Large Language Models (VLMs) have achieved remarkable success in video understanding, but the significant computational cost from processing dense frames severely limits their practical application. Existing methods alleviate this by selecting keyframes, but their greedy decision-making, combined with a decoupled evaluation of relevance and diversity, often falls into local optima and results in erroneously selecting irrelevant noise frames. To address these challenges, we propose GIFT: Global Irreplaceability Frame Targeting, a novel training-free framework that selects frames by assessing their intrinsic irreplaceability. Specifically, we first introduce Directed Diversity to quantify a frame's uniqueness conditioned on relevance, which allows us to formulate a unified irreplaceability score. Subsequently, our Budget-Aware Refinement strategy employs a adaptive iterative process that first secures a core set of frames with the highest irreplaceability, and then shifts its priority to building crucial temporal context around these selections as the budget expands. Extensive experiments demonstrate that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。