用大模型推理实现无需训练的声音定位，提升复杂场景准确性。
Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning
- 利用多模态大模型进行生成-分析-精炼三阶段推理
- 在单/多声源数据集上达到媲美训练方法的定位精度
- 适合对声音定位精度要求高、不愿训练模型的研究者
声音源定位旨在通过音频与视觉模态间的关联识别发声物体的位置。现有大多数方法依赖对比学习进行特征匹配，但缺乏显式推理与验证，在复杂声学场景中表现受限。受人类元认知过程启发，我们提出一种无需训练的声音定位框架，利用多模态大语言模型（MLLM）的内在推理能力。所提生成-分析-精炼（GAR）流程包含三个阶段：生成阶段输出初始边界框与音频分类；分析阶段通过开集角色标注与锚点投票量化音视频一致性；精炼阶段采用自适应门控机制避免不必要的调整。在单声源与多声源基准测试中均表现出色。代码已开源：https://github.com/VisualAIKHU/GAR-SSL。
原文摘要 · Abstract (English)
Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。