让视觉听觉融合更可靠，应对陌生声音环境。
Reliability-Aware Geometric Fusion for Robust Audio-Visual Navigation
- 根据音频可靠性动态调节视听信息融合
- 在未听过的声源上导航成功率提升12.7%
- 适合做复杂环境下的智能体导航研究
视听导航（AVN）要求具身智能体利用视觉和双耳音频信息向声源移动。在复杂声学环境中，双耳线索常因环境干扰而变得间歇不可靠，尤其在面对未听过的声源类别时更为显著。为此，我们提出RAVN（可靠性感知视听导航）框架，通过音频衍生的可靠性线索来调控跨模态融合，动态校准视听输入的整合。RAVN引入声学几何推理器（AGR），采用异方差高斯负对数似然目标进行训练，学习观测依赖的分布离散度作为实用的可靠性线索，从而在推理阶段无需几何标签。此外，我们设计了可靠性感知几何调制（RAGM），将学习到的可靠性线索转化为软门控机制，调节视觉特征，缓解跨模态冲突。我们在SoundSpaces数据集上的Replica和Matterport3D环境中评估RAVN，结果表明其在导航性能上持续提升，在极具挑战的未听声源设置中表现尤为稳健。
原文摘要 · Abstract (English)
Audio-Visual Navigation (AVN) requires an embodied agent to navigate toward a sound source by utilizing both vision and binaural audio. A core challenge arises in complex acoustic environments, where binaural cues become intermittently unreliable, particularly when generalizing to previously unheard sound categories. To address this, we propose RAVN (Reliability-Aware Audio-Visual Navigation), a framework that conditions cross-modal fusion on audio-derived reliability cues, dynamically calibrating the integration of audio and visual inputs. RAVN introduces an Acoustic Geometry Reasoner (AGR) that is trained with geometric proxy supervision. Using a heteroscedastic Gaussian NLL objective, AGR learns observation-dependent dispersion as a practical reliability cue, eliminating the need for geometric labels during inference. Additionally, we introduce Reliability-Aware Geometric Modulation (RAGM), which converts the learned cue into a soft gate to modulate visual features, thereby mitigating cross-modal conflicts. We evaluate RAVN on SoundSpaces using both Replica and Matterport3D environments, and the results show consistent improvements in navigation performance, with notable robustness in the challenging unheard sound setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。