用文字描述精准评分视频帧，让模型更高效理解视频内容。
From Captions to Keyframes: KeyScore for Multimodal Frame Scoring and Video-Language Understanding
- 根据字幕语义、时间代表性与上下文影响，三重信号评分帧重要性。
- 在三个基准上实现99%帧数减少，性能优于均匀采样8帧方案。
- 适合需要高效视频理解的场景，如视频检索与动作识别。
关键帧选择对高效视频理解至关重要，但现有方法常依赖启发式规则，忽略语义信息或产生冗余帧。本文提出KeyScore，一种基于字幕的帧评分方法，融合语义相似性、时间代表性与上下文缺失影响三种互补信号。该方法在大规模视频-字幕数据集上生成帧级重要性分数，可用于训练关键帧提取器或指导视频-语言模型。为此，我们还提出STACFP（时空自适应聚类），能从长视频中生成多样且紧凑的帧提案。联合使用KeyScore与STACFP，在保留关键内容的同时显著减少无意义帧，提升推理速度与准确率。在MSRVTT、MSVD、DiDeMo三个标准视频-语言基准上的实验表明，相比全帧处理，该方法可实现最高99%的帧数减少，且在视频-文本检索、关键帧提取与动作识别任务中表现超越均匀采样的8帧编码器。通过聚焦语义相关帧，本方法同时提升效率与性能，支持可扩展的、以字幕为依据的视频理解。
原文摘要 · Abstract (English)
Selecting informative keyframes is critical for efficient video understanding, yet existing approaches often rely on heuristics, ignore semantics, or produce redundant frames. We propose KeyScore, a caption-aware frame scoring method that combines three complementary signals: semantic similarity to captions, temporal representativeness, and contextual drop impact. Applied to large-scale video-caption datasets, KeyScore generates frame-level importance scores that enable training keyframe extractors or guiding video-language models. To support this, we also propose STACFP, a Spatio-Temporal Adaptive Clustering method that generates diverse and compact frame proposals across long videos. Together, KeyScore and STACFP reduce uninformative frames while preserving critical content, resulting in faster and more accurate inference. Our experiments on three standard video-language benchmarks (MSRVTT, MSVD, DiDeMo) show that combining STACFP and KeyScore enables up to 99% frame reduction compared to full-frame processing, while outperforming uniform 8-frame encoders in video-text retrieval, keyframe extraction, and action recognition tasks. By focusing on semantically relevant frames, our method enhances both efficiency and performance, enabling scalable and caption-grounded video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。