提出可任意选k帧的场景驱动关键帧方法,保持视频连贯性。
K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding
- 基于查询生成语义连贯的片段,支持任意数量关键帧选择
- 在多个长视频任务上超越现有方法,提升理解效果
- 适合需要灵活控制帧数的视频分析场景
多模态大语言模型在图像理解上表现优异,但在长视频理解中受限于上下文窗口和计算成本。均匀采样导致信息大量丢失,而现有关键帧选择方法如文本-帧检索或基于强化学习的优化通常产生稀疏且时间不连续的帧,忽视场景连贯性,且难以实现多尺度帧数选择。为此,我们提出K-frames,一种场景驱动的关键帧选择新范式,能有效保留时间连续性。K-frames不直接选取单帧,而是预测语义一致、与查询相关的视频片段,从而支持任意数量(any-k)关键帧选择以适应不同用户预算。为此,我们构建了包含20万视频亮点的PeakClips数据集,该数据集以查询为条件。在此基础上,K-frames采用三阶段渐进式课程学习:先进行两阶段监督微调以实现时间定位和关键片段感知,再通过强化学习阶段直接优化面向下游任务的场景驱动预测策略,无需额外标注。在多个主流长视频理解基准上的实验表明,K-frames提供了一种高效、可解释且即插即用的关键帧选择方案,适用于不同规模。我们的数据集和模型将公开发布。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in image understanding, but long-video are constrained by context windows and computational cost. Uniform frame sampling often leads to substantial information loss. Meanwhile existing keyframe selection methods such as text-frame retrieval or RL-based frame optimization typically yield sparse and temporally disjointed frames, overlooking scene continuity and lacking flexibility for multi-scale frame selection. To address these limitations, we introduce K-frames, a novel paradigm for scene-driven keyframe selection that preserves temporal continuity. Instead of selecting individual frames, K-frames predicts semantically coherent, query-relevant clips, which enables any-k keyframes selection to meet diverse user budgets. To achieve this approach, we first introduce PeakClips, a dataset of 200K video highlights conditioned by query. Building on this dataset, K-frames learns clip2frame selection using a three-stage progressive curriculum. It involves two Supervised Fine-Tuning stages for temporal grounding and key-clip perception, followed by a Reinforcement Learning stage that directly optimizes the scene-driven prediction policy for downstream task without further annotations. Extensive experiments on major long-video understanding benchmarks demonstrate that K-frames provides an effective, interpretable, and plug-and-play solution for keyframe selection at various scales. Our dataset and model will be available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。