用预训练模型实现个性化视频摘要,无需大量标注数据。
Personalized Video Summarization by Multimodal Video Understanding
- 基于视觉语言模型,将视频帧转为文本进行语义分析。
- 在多个数据集上优于现有无监督模型,适应性强。
- 支持多用户偏好快速扩展,适合实际应用部署。
视频摘要技术能有效提升用户对视频内容的访问与理解体验。若了解用户偏好,可从中识别出重要信息或相关内容,帮助用户快速获取所需内容或判断是否观看原视频。但针对不同视频类型和用户偏好进行适配,需大量训练数据和昂贵的人工标注。为此,我们提出一个新基准,涵盖多种用户偏好。同时,提出名为VSL的流水线,基于预训练视觉语言模型(VLMs),避免对大规模数据集进行训练。该方法以视频和字幕为输入,在场景级别进行语义分析,将视频帧转换为文本,再依据用户类型偏好筛选相关文本场景。实验表明,该方法在性能上超过当前最先进的无监督视频摘要模型,并在跨数据集适应性上优于监督查询式模型。运行时分析显示,其在扩展用户偏好和视频数量时更具实用性。
原文摘要 · Abstract (English)
Video summarization techniques have been proven to improve the overall user experience when it comes to accessing and comprehending video content. If the user's preference is known, video summarization can identify significant information or relevant content from an input video, aiding them in obtaining the necessary information or determining their interest in watching the original video. Adapting video summarization to various types of video and user preferences requires significant training data and expensive human labeling. To facilitate such research, we proposed a new benchmark for video summarization that captures various user preferences. Also, we present a pipeline called Video Summarization with Language (VSL) for user-preferred video summarization that is based on pre-trained visual language models (VLMs) to avoid the need to train a video summarization system on a large training dataset. The pipeline takes both video and closed captioning as input and performs semantic analysis at the scene level by converting video frames into text. Subsequently, the user's genre preference was used as the basis for selecting the pertinent textual scenes. The experimental results demonstrate that our proposed pipeline outperforms current state-of-the-art unsupervised video summarization models. We show that our method is more adaptable across different datasets compared to supervised query-based video summarization models. In the end, the runtime analysis demonstrates that our pipeline is more suitable for practical use when scaling up the number of user preferences and videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。