arXiv:2506.23623cs.CV2025-06CVPR被引 8

用视觉引导声音分割,提升复杂场景下的精准度。

Revisiting Audio-Visual Segmentation with Vision-Centric Transformer

  • 以视觉特征生成查询,迭代获取音视频信息
  • 在三个子集上达到新最佳性能,平均mIoU提升显著
  • 适合关注音视频联合建模与精细分割的研究者

音频-视觉分割(AVS)旨在根据音频信号定位视频中发声物体。现有方法多采用音频中心的Transformer架构,但存在两个问题:音频混合导致感知模糊,以及因视觉细节丢失而削弱密集预测能力。为此,本文提出一种视觉中心的Transformer(VCT)框架,通过视觉特征生成查询,迭代获取对应音视频信息,使查询能更好区分混叠音频中的不同发声对象,并准确勾勒其轮廓。此外,引入原型提示查询生成(PPQG)模块,利用音频原型提示和像素上下文分组,生成语义丰富且视觉清晰的查询,促进音视频信息融合。大量实验表明,该框架在AVSBench数据集三个子集上均达到新最优性能。代码已公开于https://github.com/spyflying/VCT_AVS。

原文摘要 · Abstract (English)

Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where object queries are derived from audio features. However, audio-centric Transformers suffer from two limitations: perception ambiguity caused by the mixed nature of audio, and weakened dense prediction ability due to visual detail loss. To address these limitations, we propose a new Vision-Centric Transformer (VCT) framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information, enabling queries to better distinguish between different sounding objects from mixed audio and accurately delineate their contours. Additionally, we also introduce a Prototype Prompted Query Generation (PPQG) module within our VCT framework to generate vision-derived queries that are both semantically aware and visually rich through audio prototype prompting and pixel context grouping, facilitating audio-visual information aggregation. Extensive experiments demonstrate that our VCT framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset. The code is available at https://github.com/spyflying/VCT_AVS.

音视频分割视觉引导Transformer多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。