用伪标签增强音视频片段解析,提升模型对多模态事件的识别能力。
MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing
- 通过伪标签生成跨模态新数据,增强模型对复杂事件组合的泛化能力。
- 在LLP数据集上,视觉与音频段级指标分别提升2.1%和1.2%。
- 适合关注弱监督音视频理解、多模态事件检测的研究者。
弱监督音视频视频解析(AVVP)旨在预测各模态特定事件并定位其时间边界。尽管已有显著进展,但受限于弱监督设置及模型架构缺陷,现有方法难以同时提升段级与事件级预测性能。本文提出一种伪标签增强的音视频Mamba网络(MUG),强调每个段落的独特性,并抑制异模态噪声干扰。具体地,基于前期工作标注部分伪标签,利用单模态伪标签进行跨模态随机组合生成新数据,以增强模型对多样化段级事件组合的解析能力。特征处理与交互采用音视频Mamba网络(AV-Mamba),有效提升对不同段落的感知能力,同时排除冗余模态噪声,保留共享信息。大量实验表明,MUG在LLP数据集上所有指标均优于当前最优方法(如视觉段级与音频段级指标分别提升2.1%和1.2%)。代码已公开于https://github.com/WangLY136/MUG。
原文摘要 · Abstract (English)
The weakly-supervised audio-visual video parsing (AVVP) aims to predict all modality-specific events and locate their temporal boundaries. Despite significant progress, due to the limitations of the weakly-supervised and the deficiencies of the model architecture, existing methods are lacking in simultaneously improving both the segment-level prediction and the event-level prediction. In this work, we propose a audio-visual Mamba network with pseudo labeling aUGmentation (MUG) for emphasising the uniqueness of each segment and excluding the noise interference from the alternate modalities. Specifically, we annotate some of the pseudo-labels based on previous work. Using unimodal pseudo-labels, we perform cross-modal random combinations to generate new data, which can enhance the model's ability to parse various segment-level event combinations. For feature processing and interaction, we employ a audio-visual mamba network. The AV-Mamba enhances the ability to perceive different segments and excludes additional modal noise while sharing similar modal information. Our extensive experiments demonstrate that MUG improves state-of-the-art results on LLP dataset in all metrics (e.g,, gains of 2.1% and 1.2% in terms of visual Segment-level and audio Segment-level metrics). Our code is available at https://github.com/WangLY136/MUG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。