让语音大模型像人一样精准判断声音的方位与距离。
OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models
- 用全景深度图和声学响应训练音频编码器,实现三维空间感知。
- 在真实场景中将方向误差降低11度,问答准确率提升25%。
- 适合做智能音箱、机器人听觉系统等需要空间理解的场景。
空间推理是听觉感知的基础,但现有语音大模型主要依赖无结构的双耳线索和单步推理,限制了方向与距离估计的精度及可解释性。尽管如BAT等模型已能进行空间问答,但其使用粗粒度标签(左、右、上、下)且缺乏显式几何监督,导致分辨率低、鲁棒性差。本文提出空间-声学几何编码器SAGE,训练时利用全景深度图和房间混响响应对齐双耳声学特征与三维空间结构,推理时仅需音频输入。基于此表示,构建OWL模型,结合空间锚定的思维链,对到达方向(DoA)和距离进行推理。通过从感知问答到多步推理的课程学习,实现钟表级方位角与DoA估计。为支持大规模训练与评估,我们构建并发布包含超百万样本对的BiDepth数据集,融合双耳音频、全景深度图与房间混响响应,覆盖室内与室外场景。在两个基准数据集上,相比BAT,OWL通过SAGE使平均DoA误差减少11°,空间推理问答准确率最高提升25%。
原文摘要 · Abstract (English)
Spatial reasoning is fundamental to auditory perception, yet current audio large language models (ALLMs) largely rely on unstructured binaural cues and single step inference. This limits both perceptual accuracy in direction and distance estimation and the capacity for interpretable reasoning. Recent work such as BAT demonstrates spatial QA with binaural audio, but its reliance on coarse categorical labels (left, right, up, down) and the absence of explicit geometric supervision constrain resolution and robustness. We introduce the $\textbf{Spatial-Acoustic Geometry Encoder (SAGE}$), a geometry-aware audio encoder that aligns binaural acoustic features with 3D spatial structure using panoramic depth images and room-impulse responses at training time, while requiring only audio at inference. Building on this representation, we present $\textbf{OWL}$, an ALLM that integrates $\textbf{SAGE}$ with a spatially grounded chain-of-thought to rationalize over direction-of-arrivals (DoA) and distance estimates. Through curriculum learning from perceptual QA to multi-step reasoning, $\textbf{OWL}$ supports o'clock-level azimuth and DoA estimation. To enable large-scale training and evaluation, we construct and release $\textbf{BiDepth}$, a dataset of over one million QA pairs combining binaural audio with panoramic depth images and room impulse responses across both in-room and out-of-room scenarios. Across two benchmark datasets, our new $\textbf{BiDepth}$ and the public SpatialSoundQA, $\textbf{OWL}$ reduces mean DoA error by $\textbf{11$^{\circ}$}$ through $\textbf{SAGE}$ and improves spatial reasoning QA accuracy by up to $\textbf{25}$\% over BAT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。