arXiv:2601.14594cs.CV2026-01被引 1

让视频描述更准确:智能挑选关键帧,兼顾事件重要性和时间分布。

LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioning

  • 设计可学习的帧选择器,根据事件和时间分布自动挑帧。
  • 在VDC上提升2.0%,在ICH-CC上提升超4%。
  • 适合需要高质量视频描述与问答的应用场景。

视频描述模型将帧转换为视觉标记,并用大语言模型生成描述。由于编码所有帧开销过大,当前普遍采用均匀采样,但这种方法强制均等时间覆盖,忽略了事件分布不均的问题。为此,本文提出可学习的帧选择器(LFS),旨在选取时间多样且事件相关的帧。LFS显式建模时间重要性,平衡时间多样性与事件相关性,并采用分层策略确保时间覆盖,避免帧聚集。关键在于,LFS利用冻结的视频-大模型生成的描述反馈,直接优化下游描述质量。此外,我们发现现有基准与人类认知存在差距,因此构建了基于标注者精心设计问题的ICH-CC数据集,反映人类对视频的一致理解。实验表明,LFS在两个主流社区基准及ICH-CC上均显著提升详细视频描述性能,在VDC上最高提升2.0%,在ICH-CC上超过4%。同时,使用LFS生成的增强描述也提升了视频问答表现。总体而言,LFS为精细视频描述提供了一种高效且易集成的解决方案。

原文摘要 · Abstract (English)

Video captioning models convert frames into visual tokens and generate descriptions with large language models (LLMs). Since encoding all frames is prohibitively expensive, uniform sampling is the default choice, but it enforces equal temporal coverage while ignoring the uneven events distribution. This motivates a Learnable Frame Selector (LFS) that selects temporally diverse and event-relevant frames. LFS explicitly models temporal importance to balance temporal diversity and event relevance, and employs a stratified strategy to ensure temporal coverage while avoiding clustering. Crucially, LFS leverages caption feedback from frozen video-LLMs to learn frame selection that directly optimizes downstream caption quality. Additionally, we identify the gap between existing benchmark and human's cognition. Thus, we introduce ICH-CC built from carefully designed questions by annotators that reflect human-consistent understanding of video. Experiments indicate that LFS consistently improves detailed video captioning across two representative community benchmarks and ICH-CC, achieving up to 2.0% gains on VDC and over 4% gains on ICH-CC. Moreover, we observe that enhanced captions with LFS leads to improved performance on video question answering. Overall, LFS provides an effective and easy-to-integrate solution for detailed video captioning.

视频描述帧选择大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。