用联合注意力分离音视频中的目标与干扰，提升声音定位准确率。
Improving Sound Source Localization with Joint Slot Attention on Image and Audio
- 通过竞争性槽注意力分离图像和音频中的目标与无关成分
- 在三个公开数据集上均达到最优性能，跨模态检索显著领先
- 适合需要高精度音视频对齐的场景，如智能安防、人机交互
声音源定位（SSL）任务旨在确定图像中声音的来源。由于缺乏定位标签，现有方法通常将图像和音频分别表示为单一嵌入向量，并通过对比学习进行训练。以往工作从局部图像特征中采样一个作为图像嵌入，将所有局部音频特征聚合为音频嵌入，但因输入中存在噪声和无关背景，效果不佳。本文提出一种新方法：在图像和音频上联合使用槽注意力机制，两个槽竞争性地关注图像与音频特征，将其分解为目标和非目标表示，仅使用目标部分进行对比学习。同时引入跨模态注意力匹配，进一步对齐图像与音频的局部特征。该方法在三个公开的SSL基准测试中几乎全部取得最佳表现，且在跨模态检索任务中显著优于先前方法。
原文摘要 · Abstract (English)
Sound source localization (SSL) is the task of locating the source of sound within an image. Due to the lack of localization labels, the de facto standard in SSL has been to represent an image and audio as a single embedding vector each, and use them to learn SSL via contrastive learning. To this end, previous work samples one of local image features as the image embedding and aggregates all local audio features to obtain the audio embedding, which is far from optimal due to the presence of noise and background irrelevant to the actual target in the input. We present a novel SSL method that addresses this chronic issue by joint slot attention on image and audio. To be specific, two slots competitively attend image and audio features to decompose them into target and off-target representations, and only target representations of image and audio are used for contrastive learning. Also, we introduce cross-modal attention matching to further align local features of image and audio. Our method achieved the best in almost all settings on three public benchmarks for SSL, and substantially outperformed all the prior work in cross-modal retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。