让机器理解声音的方位和语义，实现自然语言描述与空间音频的精准对应。
Learning Spatially-Aware Language and Audio Embeddings
- 通过多模态对比学习，联合建模音频语义与空间位置信息。
- 在4738小时音频上训练，文本到音频检索准确率提升2.8%。
- 适合需要空间感知的语音识别、智能音箱等场景使用。
人类能根据模糊的自然语言描述想象出声音场景，例如“狮子吼声从我身后传来”。要让机器具备类似理解能力，需掌握声音的语义属性（如狮子）与空间属性（如“身后”），并将其与声音的语义和空间特征对齐。现有音频基础模型通常仅处理非空间音频与文本配对，缺乏空间感知；而声音定位模型则受限于固定类别且仅输出绝对位置（如0.2米），无法匹配自然语言描述（如“在我旁边”）。为此，我们提出ELSA——一种基于多模态对比学习的空间感知音频与文本嵌入模型。ELSA支持非空间音频、空间音频及包含空间与语义信息的开放词汇文本描述。训练时：(a) 对三个开源音频数据集共4738小时音频及其字幕进行空间增强；(b) 设计编码器同时捕捉非空间音频的语义，以及空间音频的语义与空间属性。ELSA在语义检索和3D声源定位任务中均表现优异，相较基线，音频到文本与文本到音频的R@1提升2.8%，3D声源定位平均绝对误差降低11.6°。
原文摘要 · Abstract (English)
Humans can picture a sound scene given an imprecise natural language description. For example, it is easy to imagine an acoustic environment given a phrase like "the lion roar came from right behind me!". For a machine to have the same degree of comprehension, the machine must know what a lion is (semantic attribute), what the concept of "behind" is (spatial attribute) and how these pieces of linguistic information align with the semantic and spatial attributes of the sound (what a roar sounds like when its coming from behind). State-of-the-art audio foundation models which learn to map between audio scenes and natural textual descriptions, are trained on non-spatial audio and text pairs, and hence lack spatial awareness. In contrast, sound event localization and detection models are limited to recognizing sounds from a fixed number of classes, and they localize the source to absolute position (e.g., 0.2m) rather than a position described using natural language (e.g., "next to me"). To address these gaps, we present ELSA a spatially aware-audio and text embedding model trained using multimodal contrastive learning. ELSA supports non-spatial audio, spatial audio, and open vocabulary text captions describing both the spatial and semantic components of sound. To train ELSA: (a) we spatially augment the audio and captions of three open-source audio datasets totaling 4,738 hours of audio, and (b) we design an encoder to capture the semantics of non-spatial audio, and the semantics and spatial attributes of spatial audio using contrastive learning. ELSA is competitive with state-of-the-art for both semantic retrieval and 3D source localization. In particular, ELSA achieves +2.8% mean audio-to-text and text-to-audio R@1 above the baseline, and outperforms by -11.6° mean-absolute-error in 3D source localization over the baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。