用扩散模型替代传统序列模型,实现高效通用音频理解。
DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding
- 采用扩散架构与双适配器,提升音频表征能力。
- 在三个数据集上超越前代模型,接近主流自回归模型性能。
- 全开源训练,适合追求高效音频建模的研究者。
自回归大型音频语言模型(LALMs)如 Qwen-2.5-Omni 在音频理解与交互任务中表现优异,但其扩展需大量数据与计算资源,且严格串行解码限制了推理效率。扩散型大语言模型(dLLMs)被证明可高效利用有限训练数据,先前的 DIFFA 工作表明,将自回归主干替换为扩散模型能显著提升音频理解能力,但仅限于概念验证规模,缺乏大规模指令微调、偏好对齐与实用解码方案。本文提出 DIFFA-2,一种面向通用音频理解的实用扩散型 LALM。它升级了语音编码器,引入语义与声学双适配器,并通过四阶段课程训练:结合语义与声学对齐、大规模监督微调及方差缩减的偏好优化,仅使用完全开源语料库。在 MMSU、MMAU 与 MMAR 数据集上的实验显示,DIFFA-2 持续优于 DIFFA,且在实际训练预算下具备与强自回归模型相当的竞争力,验证了扩散建模作为大规模音频理解主干的可行性。代码已开源。
原文摘要 · Abstract (English)
Autoregressive (AR) large audio language models (LALMs) such as Qwen-2.5-Omni have achieved strong performance on audio understanding and interaction, but scaling them remains costly in data and computation, and strictly sequential decoding limits inference efficiency. Diffusion large language models (dLLMs) have recently been shown to make effective use of limited training data, and prior work on DIFFA indicates that replacing an AR backbone with a diffusion counterpart can substantially improve audio understanding under matched settings, albeit at a proof-of-concept scale without large-scale instruction tuning, preference alignment, or practical decoding schemes. We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding. DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum that combines semantic and acoustic alignment, large-scale supervised fine-tuning, and variance-reduced preference optimization, using only fully open-source corpora. Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA and is competitive to strong AR LALMs under practical training budgets, supporting diffusion-based modeling is a viable backbone for large-scale audio understanding. Our code is available at https://github.com/NKU-HLT/DIFFA.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。