让声音指导视觉分割,提升对未知物体的识别能力。
Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

- 用声音动态调整视觉先验,实现音视频协同定位
- 在未见类别上性能超越现有方法,尤其擅长处理新物体
- 适合研究跨模态语义分割与开放词汇学习的开发者
开放词汇音视频语义分割(OV-AVSS)旨在对开放类别中的发声物体进行像素级分割。以往方法依赖无类别的前景定义,将语义差异大的物体归为同一正样本集,导致模型学习不稳定的听觉特征并生成不可靠建议。为此,本文提出类别特定的目标重构,并设计新型声学引导代价学习(AGCL)框架,将静态、音频无关的视觉-文本先验转化为动态、音频引导的代价表示。针对类别内听觉特性发现,引入音频调制代价生成(AMCG)与音频引导时序聚合(AGTA)模块,通过低侵入式音频注入机制实现帧级发声区域突出和视频级时序优化。针对类别间干扰项区分,提出协同干扰项挖掘(SDM)策略,有选择地惩罚声学与语义混淆的负类别,以学习更判别性的决策边界。在AVSBench-OV数据集上的大量实验表明,该方法显著优于当前最优方案,尤其在未见类别上表现突出。代码已开源。
原文摘要 · Abstract (English)
Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。