用CLIP实现无需文本的自监督声源定位,效果超越现有方法。
Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization
- 将音频转为兼容CLIP文本编码器的标记,生成音频嵌入。
- 通过对比学习对齐音视频特征,定位发声区域更完整紧凑。
- 支持零样本泛化,适合多场景声源定位任务。
大规模视觉-语言模型展现出强大的多模态对齐与泛化能力。其中,CLIP是最成功的范例之一。本文将CLIP拓展至声源定位任务,提出一种无需显式文本输入的自监督方法。我们构建一个框架,将音频映射为适配CLIP文本编码器的标记,生成音频驱动的嵌入表示。这些嵌入用于生成发声区域掩码,从中提取视觉特征,并通过对比音视频对应关系目标进行对齐。实验表明,预训练多模态基础模型的对齐知识使我们的方法能生成更完整、更紧凑的发声物体定位结果。我们进一步提出基于LLM引导的扩展方案,在训练中蒸馏对象感知的音视频场景理解以增强对齐。在五个不同任务上的大量实验表明,本方法所有变体均优于当前最先进方法,并在零样本设置下表现出强泛化能力。
原文摘要 · Abstract (English)
Large-scale vision-language models demonstrate strong multimodal alignment and generalization across diverse tasks. Among them, CLIP stands out as one of the most successful approaches. In this work, we extend the application of CLIP to sound source localization, proposing a self-supervised method operates without explicit text input. We introduce a framework that maps audios into tokens compatible with CLIP's text encoder, producing audio-driven embeddings. These embeddings are used to generate sounding region masks, from which visual features are extracted and aligned with the audio embeddings through a contrastive audio-visual correspondence objective. Our findings show that alignment knowledge of pre-trained multimodal foundation model enables our method to generate more complete and compact localization for sounding objects. We further propose an LLM-guided extension that distills object-aware audio-visual scene understanding into the model during training to enhance alignment. Extensive experiments across five diverse tasks demonstrate that our method, in all variants, outperforms state-of-the-art approaches and achieves strong generalization in zero-shot settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。