用音频引导池化,从现成模型中解锁声音定位能力
Unlocking Spatial Grounding in Large Audio-Visual Retrieval models

- 用音频信息指导中间视觉特征的池化,恢复被丢弃的空间细节
- 在AVSBench和AVATAR上达最新水平,后者性能近乎翻倍
- 无需重新训练,可复用现有大模型实现检索与定位统一
弱监督为音视频声源定位提供了实用方案,因密集空间标注难以大规模获取。该任务仍具挑战性,因模型需在无像素级监督下,从时序对齐的音视频数据中定位声源。近期大规模音视频检索模型虽经空前规模训练,编码了丰富的多模态结构,但其潜在表示即使优化于全局对齐,仍可实现细粒度空间定位。尽管高层特征因全局池化逐步丢失空间细节,中间视觉标记仍保留高度结构化空间信息。为此,我们提出LAIP(基于音频引导池化的定位),采用轻量级音频引导空间池化(AiSP)替换标准全局聚合模块。通过使用帧对齐音频查询中间视觉标记,LAIP恢复了原本被冻结检索流程丢弃的局部空间信息。该方法在AVSBench和AVATAR上达到最先进性能,尤其在后者上几乎实现性能翻倍。结果表明,精确定位无需从头学习,而是可从现有检索表示中解锁,为检索与定位任务提供统一路径。
原文摘要 · Abstract (English)
Weak supervision sets a practical regime for audio-visual sound source localization as dense spatial annotations are costly to obtain at scale. The task, however, remains challenging, as models must locate sound sources from temporally aligned audio-visual data without pixel-level supervision. Recent large-scale audio-visual retrieval models, trained at unprecedented scale, encode rich multimodal structure. We show their latent representations, though optimized for global alignment, can nonetheless enable fine-grained spatial grounding. While spatial detail is progressively lost in the upper layers of retrieval backbones due to global pooling, intermediate visual tokens retain highly structured spatial information. To exploit this, we introduce LAIP (\emph{Localization via Audio-Informed Pooling}), a framework that employs a lightweight \emph{Audio-informed Spatial Pooling} (AiSP) to replace the standard global aggregation module. By using frame-aligned audio to query intermediate visual tokens, LAIP recovers localized spatial information that is otherwise discarded by the frozen retrieval pipeline. Our approach achieves state-of-the-art performance on AVSBench and AVATAR, nearly doubling previous results on the latter. These findings prove that accurate localization does not need to be learned from scratch; instead, it can be unlocked from existing retrieval representations, providing a unified path for both retrieval and localization tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。