从长视频中选出关键片段,用少量片段生成完整摘要。
Minimal Clips, Maximum Salience: Long Video Summarization via Key Moment Extraction
- 用轻量模型为视频分段生成视觉描述,再由大模型选关键片段。
- 仅用不到6%的视频片段,就能还原完整电影摘要。
- 计算成本低,适合高效分析长视频内容。
视觉语言模型(VLM)能处理越来越长的视频,但重要视觉信息常在上下文中丢失。本文提出一种关键片段选择方法,将视频分割为短片段,使用轻量级视频描述模型生成每个片段的紧凑视觉描述,再交由大语言模型(LLM)选出最相关的K个片段用于多模态摘要。我们在MovieSum数据集上评估,该方法基于人工标注的剧本和摘要自动生成参考片段。结果表明,这些参考片段(不足影片总长度的6%)已足以构建完整多模态摘要。相比随机选取片段,本方法在保持接近参考片段摘要性能的同时,捕获了更多相关视频信息,且通过轻量级描述模型维持了较低计算开销。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) are able to process increasingly longer videos. Yet, important visual information is easily lost throughout the entire context and missed by VLMs. Also, it is important to design tools that enable cost-effective analysis of lengthy video content. In this paper, we propose a clip selection method that targets key video moments to be included in a multimodal summary. We divide the video into short clips and generate compact visual descriptions of each using a lightweight video captioning model. These are then passed to a large language model (LLM), which selects the K clips containing the most relevant visual information for a multimodal summary. We evaluate our approach on reference clips for the task, automatically derived from full human-annotated screenplays and summaries in the MovieSum dataset. We further show that these reference clips (less than 6% of the movie) are sufficient to build a complete multimodal summary of the movies in MovieSum. Using our clip selection method, we achieve a summarization performance close to that of these reference clips while capturing substantially more relevant video information than random clip selection. Importantly, we maintain low computational cost by relying on a lightweight captioning model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。