用智能代理自动剪辑视频,保留精华并讲好故事。
Agent-based Video Trimming
- 分三阶段:先描述视频片段,再过滤差内容,最后组装成连贯叙事。
- 在多个数据集上比现有方法更准地识别精彩片段,准确率更高。
- 适合需要高效剪辑长视频的创作者或内容平台使用。
随着信息获取日益便捷,用户生成视频长度不断增长,给观众筛选有价值内容带来负担。尽管亮点检测、片段检索和视频摘要已有显著进展,现有方法多聚焦于选择特定时间段,忽视了片段间的关联性及排列组合的可能性。本文提出新任务视频剪辑(Video Trimming, VT),旨在识别冗余画面、选取高价值片段,并将其组合成逻辑连贯的最终视频。为此,我们提出基于智能体的视频剪辑框架(Agent-based Video Trimming, AVT),包含三个阶段:视频结构化、片段过滤与故事编排。具体地,采用视频描述代理将视频切片转化为结构化文本;通过过滤模块根据片段结构信息动态剔除低质量内容;再由视频排列代理选择并整合有效片段形成连贯叙事。为评估效果,我们设计视频评估代理,与人工评估并行开展。此外,我们基于互联网原始用户视频构建了一个新的视频剪辑基准数据集。实验表明,AVT在用户研究中获得更优评价,在YouTube Highlights、TVSum及自建数据集上的亮点检测任务中,mAP和精度均优于现有方法。代码与模型已公开于 https://ylingfeng.github.io/AVT。
原文摘要 · Abstract (English)
As information becomes more accessible, user-generated videos are increasing in length, placing a burden on viewers to sift through vast content for valuable insights. This trend underscores the need for an algorithm to extract key video information efficiently. Despite significant advancements in highlight detection, moment retrieval, and video summarization, current approaches primarily focus on selecting specific time intervals, often overlooking the relevance between segments and the potential for segment arranging. In this paper, we introduce a novel task called Video Trimming (VT), which focuses on detecting wasted footage, selecting valuable segments, and composing them into a final video with a coherent story. To address this task, we propose Agent-based Video Trimming (AVT), structured into three phases: Video Structuring, Clip Filtering, and Story Composition. Specifically, we employ a Video Captioning Agent to convert video slices into structured textual descriptions, a Filtering Module to dynamically discard low-quality footage based on the structured information of each clip, and a Video Arrangement Agent to select and compile valid clips into a coherent final narrative. For evaluation, we develop a Video Evaluation Agent to assess trimmed videos, conducting assessments in parallel with human evaluations. Additionally, we curate a new benchmark dataset for video trimming using raw user videos from the internet. As a result, AVT received more favorable evaluations in user studies and demonstrated superior mAP and precision on the YouTube Highlights, TVSum, and our own dataset for the highlight detection task. The code and models are available at https://ylingfeng.github.io/AVT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。