arXiv:2412.08357cs.CV2024-12AAAI被引 5

用生成模型降低主观标注噪声,自动提炼视频关键内容

Video Summarization using Denoising Diffusion Probabilistic Model

  • 基于去噪扩散模型构建生成式摘要框架,从概率分布学习
  • 在多个数据集上优于传统判别方法,抗噪声能力强
  • 适合需要鲁棒性摘要的视频分析场景,如自动剪辑

视频摘要旨在消除视觉冗余,保留视频关键信息以生成简洁全面的概要。现有方法多采用判别模型预测帧重要性得分,但易受不同标注者主观差异导致的标注不一致影响。本文提出一种生成式视频摘要框架,从概率分布角度学习生成摘要,有效降低主观标注噪声干扰。具体地,我们基于去噪扩散概率模型(DDPM)提出新型扩散摘要方法,通过噪声预测学习训练数据的概率分布,并通过迭代去噪生成摘要。该方法对主观标注噪声更具鲁棒性,且比判别方法更不易过拟合,具备更强泛化能力。此外,为在有限数据下训练DDPM,我们采用无监督视频摘要模型实现早期去噪过程。在TVSum、SumMe和FPVSum等多个数据集上的大量实验验证了方法的有效性。

原文摘要 · Abstract (English)

Video summarization aims to eliminate visual redundancy while retaining key parts of video to construct concise and comprehensive synopses. Most existing methods use discriminative models to predict the importance scores of video frames. However, these methods are susceptible to annotation inconsistency caused by the inherent subjectivity of different annotators when annotating the same video. In this paper, we introduce a generative framework for video summarization that learns how to generate summaries from a probability distribution perspective, effectively reducing the interference of subjective annotation noise. Specifically, we propose a novel diffusion summarization method based on the Denoising Diffusion Probabilistic Model (DDPM), which learns the probability distribution of training data through noise prediction, and generates summaries by iterative denoising. Our method is more resistant to subjective annotation noise, and is less prone to overfitting the training data than discriminative methods, with strong generalization ability. Moreover, to facilitate training DDPM with limited data, we employ an unsupervised video summarization model to implement the earlier denoising process. Extensive experiments on various datasets (TVSum, SumMe, and FPVSum) demonstrate the effectiveness of our method.

视频摘要扩散模型生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。