通过音频引导视觉对齐,提升音视频分割的鲁棒性。
Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment
- 按音频响应分组对齐音视频特征,聚焦相关区域。
- 在AVA数据集上达到81.3% mIoU,优于现有方法。
- 适合处理声音状态频繁变化的复杂场景。
基于音视频线索准确定位可听对象是音视频分割的核心目标。以往方法多侧重空间或时间上的多模态建模,却忽视了音视频对应模糊的问题,如视觉相似但声音不同的物体、以及声音状态频繁切换的情况,导致难以可靠关联音视频线索,引发过分割或欠分割。为此,我们提出一种新框架,包含两个核心组件:音频引导模态对齐(AMA)模块与不确定性估计(UE)模块。AMA不采用全局注意力盲目关联音视频线索,而是将音视频交互限制在多个组内,并根据特征对音频的响应程度,将其融合为紧凑表示,有效引导模型关注音频相关的区域。结合对比学习,AMA通过将强音频响应特征视为正样本、弱响应为负样本,进一步区分发声区与静音区。此外,UE融合空间与时间信息,识别由声音状态频繁变化引起的高不确定性区域,通过降低这些区域的预测置信度减少误差。实验表明,本方法在复杂场景下显著优于现有最先进方法,在AVA数据集上达到81.3% mIoU。
原文摘要 · Abstract (English)
Accurately localizing audible objects based on audio-visual cues is the core objective of audio-visual segmentation. Most previous methods emphasize spatial or temporal multi-modal modeling, yet overlook challenges from ambiguous audio-visual correspondences such as nearby visually similar but acoustically different objects and frequent shifts in objects' sounding status. Consequently, they may struggle to reliably correlate audio and visual cues, leading to over- or under-segmentation. To address these limitations, we propose a novel framework with two primary components: an audio-guided modality alignment (AMA) module and an uncertainty estimation (UE) module. Instead of indiscriminately correlating audio-visual cues through a global attention mechanism, AMA performs audio-visual interactions within multiple groups and consolidates group features into compact representations based on their responsiveness to audio cues, effectively directing the model's attention to audio-relevant areas. Leveraging contrastive learning, AMA further distinguishes sounding regions from silent areas by treating features with strong audio responses as positive samples and weaker responses as negatives. Additionally, UE integrates spatial and temporal information to identify high-uncertainty regions caused by frequent changes in sound state, reducing prediction errors by lowering confidence in these areas. Experimental results demonstrate that our approach achieves superior accuracy compared to existing state-of-the-art methods, particularly in challenging scenarios where traditional approaches struggle to maintain reliable segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。