用解耦表征提升大模型对空间音频的推理能力，参数仅增0.2%。
DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models
- 基于SpatialAST设计解耦空间音频表征，分离声源类型、方向与距离信息。
- 在SpatialSoundQA上性能超越原模型，实现更精准的空间音频理解。
- 适合需要高精度空间音频推理的多模态系统研发者。
利用大语言模型进行空间音频推理需依赖音频编码器获取声学嵌入。该编码器须同时捕捉声音事件类型、声源方向与距离等独立信息，单一编码器难以兼顾，导致性能下降。本文提出DSpAST，基于SpatialAST的新型音频编码器，在仅增加0.2%参数量的前提下，学习解耦的空间音频表征。在SpatialSoundQA数据集上的实验表明，结合大语言模型系统BAT，DSpAST显著优于原始SpatialAST，验证了其在空间音频推理任务中的有效性。
原文摘要 · Abstract (English)
Reasoning about spatial audio with large language models requires a spatial audio encoder as an acoustic front-end to obtain audio embeddings for further processing. Such an encoder needs to capture all information required to detect the type of sound events, as well as the direction and distance of their corresponding sources. Accomplishing this with a single audio encoder is demanding as the information required for each of these tasks is mostly independent of each other. As a result, the performance obtained with a single encoder is often worse than when using task-specific audio encoders. In this work, we present DSpAST, a novel audio encoder based on SpatialAST that learns disentangled representations of spatial audio while having only 0.2% additional parameters. Experiments on SpatialSoundQA with the spatial audio reasoning system BAT demonstrate that DSpAST significantly outperforms SpatialAST.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。