arXiv:2506.10807cs.CV2025-06被引 9

用自然语言指令零样本生成视频摘要,无需训练数据即可精准控制内容重点。

Prompts to Summaries: Zero-Shot Language-Guided Video Summarization with Large Language and Video Models

  • 通过大语言模型解析文本指令,动态评估视频场景重要性。
  • 在SumMe和TVSum上超越所有无监督方法,接近有监督模型性能。
  • 适合需要灵活、低成本视频摘要的科研与产品团队使用。

视频数据爆炸式增长催生了无需训练数据、可由用户自然语言控制的摘要工具需求。现有方法或依赖特定领域数据限制泛化能力,或无法融入用户意图。本文提出首个零样本、文本可查询的视频摘要框架Prompts-to-Summaries:利用现成视频-语言模型(VidLMs)生成场景描述,再通过大语言模型(LLMs)判断其重要性,无需训练数据即超越无监督方法,并达到与有监督方法相当的水平。该流程包括:(i) 视频分段;(ii) 采用内存高效的批量提示方案生成场景描述,可在单张GPU上处理数小时视频;(iii) 使用定制提示让LLM评分场景重要性;(iv) 通过新提出的时序一致性与新颖性指标,将得分传播至帧级以实现细粒度重要性评估。在SumMe和TVSum数据集上,该方法超越所有以往数据密集型无监督方法,在查询聚焦视频摘要基准上表现优异,而对比方法需帧级标注。我们还发布了VidSum-Reason数据集,包含长尾概念与多步推理任务,为本框架提供首个挑战性基线。结果表明,预训练多模态模型结合合理提示与得分传播机制,可构建通用且文本可查询的视频摘要基础。

原文摘要 · Abstract (English)

The explosive growth of video data intensified the need for flexible user-controllable summarization tools that operate without training data. Existing methods either rely on domain-specific datasets, limiting generalization, or cannot incorporate user intent expressed in natural language. We introduce Prompts-to-Summaries: the first zero-shot, text-queryable video-summarizer that converts off-the-shelf video-language models (VidLMs) captions into user-guided skims via large-language-models (LLMs) judging, without the use of training data, beating unsupervised and matching supervised methods. Our pipeline (i) segments video into scenes, (ii) produces scene descriptions with a memory-efficient batch prompting scheme that scales to hours on a single GPU, (iii) scores scene importance with an LLM via tailored prompts, and (iv) propagates scores to frames using new consistency (temporal coherence) and uniqueness (novelty) metrics for fine-grained frame importance. On SumMe and TVSum, our approach surpasses all prior data-hungry unsupervised methods and performs competitively on the Query-Focused Video Summarization benchmark, where the competing methods require supervised frame-level importance. We release VidSum-Reason, a query-driven dataset featuring long-tailed concepts and multi-step reasoning, where our framework serves as the first challenging baseline. Overall, we demonstrate that pretrained multi-modal models, when orchestrated with principled prompting and score propagation, provide a powerful foundation for universal, text-queryable video summarization.

视频摘要零样本大模型自然语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。