arXiv:2504.15921cs.CV2025-04被引 1

用大模型自动生成长视频摘要,无需人工标注。

ViSMaP: Unsupervised Hour-long Video Summarisation by Meta-Prompting

  • 通过元提示迭代优化,用短视频描述生成长视频伪摘要。
  • 在多个数据集上达到监督模型水平,且跨领域泛化能力强。
  • 适合需要低成本长视频理解的场景,如教育、新闻回放。

我们提出ViSMaP:一种基于元提示的无监督长时视频摘要方法,可处理时长一小时以上的视频。现有视频理解模型多针对预分割的短片段有效,但在长视频中因事件稀疏分布且未预分割而表现不佳。传统长视频理解依赖昂贵的有监督层级训练,需大量标注,成本高且易不一致。ViSMaP通过大语言模型(LLM)利用短视频片段描述生成长视频的伪摘要作为训练数据,避免对长视频进行人工标注。具体采用元提示策略,迭代生成并优化伪摘要:三个连续运行的LLM分别负责生成摘要、评估质量、优化生成提示。该迭代过程对提升摘要质量至关重要,因生成效果高度依赖初始提示,且不同视频差异大。我们在多个数据集上进行评估,结果表明ViSMaP性能接近全监督先进模型,并具备跨领域泛化能力。代码将在发表后公开。

原文摘要 · Abstract (English)

We introduce ViSMap: Unsupervised Video Summarisation by Meta Prompting, a system to summarise hour long videos with no-supervision. Most existing video understanding models work well on short videos of pre-segmented events, yet they struggle to summarise longer videos where relevant events are sparsely distributed and not pre-segmented. Moreover, long-form video understanding often relies on supervised hierarchical training that needs extensive annotations which are costly, slow and prone to inconsistency. With ViSMaP we bridge the gap between short videos (where annotated data is plentiful) and long ones (where it's not). We rely on LLMs to create optimised pseudo-summaries of long videos using segment descriptions from short ones. These pseudo-summaries are used as training data for a model that generates long-form video summaries, bypassing the need for expensive annotations of long videos. Specifically, we adopt a meta-prompting strategy to iteratively generate and refine creating pseudo-summaries of long videos. The strategy leverages short clip descriptions obtained from a supervised short video model to guide the summary. Each iteration uses three LLMs working in sequence: one to generate the pseudo-summary from clip descriptions, another to evaluate it, and a third to optimise the prompt of the generator. This iteration is necessary because the quality of the pseudo-summaries is highly dependent on the generator prompt, and varies widely among videos. We evaluate our summaries extensively on multiple datasets; our results show that ViSMaP achieves performance comparable to fully supervised state-of-the-art models while generalising across domains without sacrificing performance. Code will be released upon publication.

视频摘要无监督学习大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。