arXiv:2510.13891cs.LGcs.AI2025-10被引 12

提出可任意选k帧的场景驱动关键帧方法,保持视频连贯性。

K-frames: Scene-Driven Any-k Keyframe Selection for long video understanding

  • 基于查询生成语义连贯的片段,支持任意数量关键帧选择
  • 在多个长视频任务上超越现有方法,提升理解效果
  • 适合需要灵活控制帧数的视频分析场景

多模态大语言模型在图像理解上表现优异,但在长视频理解中受限于上下文窗口和计算成本。均匀采样导致信息大量丢失,而现有关键帧选择方法如文本-帧检索或基于强化学习的优化通常产生稀疏且时间不连续的帧,忽视场景连贯性,且难以实现多尺度帧数选择。为此,我们提出K-frames,一种场景驱动的关键帧选择新范式,能有效保留时间连续性。K-frames不直接选取单帧,而是预测语义一致、与查询相关的视频片段,从而支持任意数量(any-k)关键帧选择以适应不同用户预算。为此,我们构建了包含20万视频亮点的PeakClips数据集,该数据集以查询为条件。在此基础上,K-frames采用三阶段渐进式课程学习:先进行两阶段监督微调以实现时间定位和关键片段感知,再通过强化学习阶段直接优化面向下游任务的场景驱动预测策略,无需额外标注。在多个主流长视频理解基准上的实验表明,K-frames提供了一种高效、可解释且即插即用的关键帧选择方案,适用于不同规模。我们的数据集和模型将公开发布。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in image understanding, but long-video are constrained by context windows and computational cost. Uniform frame sampling often leads to substantial information loss. Meanwhile existing keyframe selection methods such as text-frame retrieval or RL-based frame optimization typically yield sparse and temporally disjointed frames, overlooking scene continuity and lacking flexibility for multi-scale frame selection. To address these limitations, we introduce K-frames, a novel paradigm for scene-driven keyframe selection that preserves temporal continuity. Instead of selecting individual frames, K-frames predicts semantically coherent, query-relevant clips, which enables any-k keyframes selection to meet diverse user budgets. To achieve this approach, we first introduce PeakClips, a dataset of 200K video highlights conditioned by query. Building on this dataset, K-frames learns clip2frame selection using a three-stage progressive curriculum. It involves two Supervised Fine-Tuning stages for temporal grounding and key-clip perception, followed by a Reinforcement Learning stage that directly optimizes the scene-driven prediction policy for downstream task without further annotations. Extensive experiments on major long-video understanding benchmarks demonstrate that K-frames provides an effective, interpretable, and plug-and-play solution for keyframe selection at various scales. Our dataset and model will be available.

视频理解关键帧多模态强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。