将空间音频融入多模态大模型，提升声音定位与场景理解能力。
Spatial-Omni: Spatial Audio Understanding Integration in Multimodal LLMs via FOA Encoding
- 用FOA编码注入空间音频为独立模态，不改动原有音频编码器。
- 在40万段空间音频上训练，16项任务平均性能超现有模型。
- 适合做智能音箱、虚拟现实等需要空间听觉理解的场景。
当前多模态大语言模型主要处理单声道音频，忽略了空间音频中的方位信息，难以实现声音定位、空间关系推理和场景理解。本文提出Spatial-Omni，一种轻量级方法，通过SO-Encoder将一阶全向声场（FOA）空间音频作为独立模态注入现有Omni LLM，无需修改其原始音频编码器。SO-Encoder以少量额外上下文开销生成空间令牌，并通过高效分阶段训练提升空间音频理解能力。为支持训练与评估，构建了SO-Dataset、SO-QA和SO-Bench，包含40万段FOA空间音频片段和210万对空间问答数据，覆盖16个子任务，涵盖基础检测、位置估计、空间关系理解与复杂空间推理。实验表明，Spatial-Omni在空间音频理解任务上优于现有开源大音频语言模型（LALMs）和Omni LLM模型，同时保持合理的通用音频理解水平。代码与数据已公开于https://github.com/dieKarotte/Spatial-Omni。
原文摘要 · Abstract (English)
Recent multimodal large language models mainly process audio as monaural signals, thereby discarding the spatial cues contained in spatial audio for sound localization, spatial relation reasoning, and spatial scene understanding. We propose Spatial-Omni, a lightweight method that implements SO-Encoder to inject First-Order Ambisonics (FOA) spatial audio into existing Omni LLMs as an independent modality, without modifying their original audio encoders. SO-Encoder provides spatial tokens with limited additional context cost and improves spatial audio understanding through efficient staged training. To support training and evaluation, we construct SO-Dataset, SO-QA, and SO-Bench from open-source data, real recordings, and simulations, containing 400K FOA spatial audio clips and 2.1M spatial question answering pairs. SO-Bench covers 16 spatial audio understanding subtasks, including basic detection and location estimation, spatial relation understanding, and complex spatial reasoning. Experiments show that Spatial-Omni outperforms existing open-source Large Audio-Language Models (LALMs) and Omni LLM models on spatial audio understanding tasks while retaining a reasonable level of general audio understanding. Code and data are available at https://github.com/dieKarotte/Spatial-Omni.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。