让语音大模型同时听懂语言和声音细节，提升感知能力
Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs
- 用统一的结构化标签框住语音中的语言、语气、事件三类信息
- 在保持推理能力的同时，细粒度感知性能提升10.9%
- 适合想提升语音模型听觉理解能力的研究者
近期语音大模型在复杂推理任务中表现优异，但在细粒度声学感知上持续表现不佳。我们发现根源在于以语音识别（ASR）为中心的训练方式，它虽提供精确的语言目标，却隐式教导模型将语气特征和声学事件视为噪声而忽略。为此，我们提出统一音频框架（UAS），将音频信息组织为转录、副语言特征与非语言事件三个显式组件，采用统一JSON格式实现全面声学覆盖，同时保持音文对齐以支持推理。该方法在离散与连续架构上均有效。在MMSU、MMAR和MMAU数据集上的实验表明，UAS-Audio模型在相同规模下比现有最优模型在细粒度感知上提升10.9%，且推理能力未受损。代码与模型已公开。
原文摘要 · Abstract (English)
Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion: while excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception. We attribute this gap to a fundamental limitation of ASR-centric training, which provides precise linguistic targets but implicitly teaches models to suppress paralinguistic cues and acoustic events as noise. To address this, we propose Unified Audio Schema (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components -- Transcription, Paralinguistics, and Non-linguistic Events -- within a unified JSON format. This design achieves comprehensive acoustic coverage without sacrificing the tight audio-text alignment that enables reasoning. We validate the effectiveness of this supervision strategy by applying it to both discrete and continuous AudioLLM architectures. Extensive experiments on MMSU, MMAR, and MMAU demonstrate that UAS-Audio yields consistent improvements, boosting fine-grained perception by 10.9% on MMSU over the same-size state-of-the-art models while preserving robust reasoning capabilities. Our code and model are publicly available at https://github.com/Tencent/Unified_Audio_Schema.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。