arXiv:2505.09615cs.CVcs.SD2025-05CVPR被引 3

弱监督下提升音视频事件定位准确率,通过不确定性加权优化伪标签。

UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing

  • 基于不确定性加权生成伪标签,缓解无段间依赖问题。
  • 在两个数据集上超越现有方法,平均性能提升超5个百分点。
  • 适合音视频分析、弱监督学习研究者参考。

音视频视频解析(AVVP)旨在定位单模态事件(仅出现在视觉或音频中)和多模态事件(同时出现在两模态中)。然而,标注所有事件的类别及其起止时间成本高昂,限制了技术扩展性,除非能在仅提供模态无关视频级标签的弱监督设置下训练。现有方法尝试生成段级伪标签以指导模型,但缺乏段间依赖建模,且倾向于忽略某段中缺失的标签,导致性能受限。本文提出不确定性加权弱监督音视频解析(UWAV),通过考虑伪标签的不确定性并引入特征混合训练正则化,改善训练效果。实验证明,UWAV在多个指标上优于当前最佳方法,在两个不同数据集上表现优异,验证了其有效性与泛化能力。

原文摘要 · Abstract (English)

Audio-Visual Video Parsing (AVVP) entails the challenging task of localizing both uni-modal events (i.e., those occurring exclusively in either the visual or acoustic modality of a video) and multi-modal events (i.e., those occurring in both modalities concurrently). Moreover, the prohibitive cost of annotating training data with the class labels of all these events, along with their start and end times, imposes constraints on the scalability of AVVP techniques unless they can be trained in a weakly-supervised setting, where only modality-agnostic, video-level labels are available in the training data. To this end, recently proposed approaches seek to generate segment-level pseudo-labels to better guide model training. However, the absence of inter-segment dependencies when generating these pseudo-labels and the general bias towards predicting labels that are absent in a segment limit their performance. This work proposes a novel approach towards overcoming these weaknesses called Uncertainty-weighted Weakly-supervised Audio-visual Video Parsing (UWAV). Additionally, our innovative approach factors in the uncertainty associated with these estimated pseudo-labels and incorporates a feature mixup based training regularization for improved training. Empirical results show that UWAV outperforms state-of-the-art methods for the AVVP task on multiple metrics, across two different datasets, attesting to its effectiveness and generalizability.

音视频解析弱监督伪标签多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。