首个基于扩散模型的语音理解大模型,能听懂口语并推理。
DIFFA: Large Language Diffusion Models Can Listen and Understand
- 用扩散模型+轻量双适配器融合语音与语言理解
- 仅用960小时语音数据就达到领先性能
- 适合想探索非自回归语音模型的研究者
大语言模型在文本和多模态领域表现出色,而基于扩散的语言模型作为自回归范式的替代方案,具备更好的可控性、双向上下文建模和鲁棒生成能力。然而其在音频领域的应用仍不充分。本文提出DIFFA,首个基于扩散的大型音频-语言模型,用于实现口语理解。DIFFA结合冻结的扩散语言模型与轻量级双适配器架构,连接语音理解与自然语言推理。采用两阶段训练:第一阶段通过语音识别目标对齐语义表征;第二阶段利用大语言模型自动生成合成语音-图文对,学习指令遵循能力。尽管仅在960小时语音识别数据和127小时合成指令数据上训练,DIFFA在MMSU、MMAU和VoiceBench等多个基准测试中表现优异,超越多个自回归开源基线。结果表明,扩散模型在高效、可扩展的音频理解方面具有潜力,为语音驱动AI开辟新方向。代码将公开于https://github.com/NKU-HLT/DIFFA.git。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have shown remarkable capabilities across textual and multimodal domains. In parallel, diffusion-based language models have emerged as a promising alternative to the autoregressive paradigm, offering improved controllability, bidirectional context modeling, and robust generation. However, their application to the audio modality remains underexplored. In this work, we introduce \textbf{DIFFA}, the first diffusion-based large audio-language model designed to perform spoken language understanding. DIFFA integrates a frozen diffusion language model with a lightweight dual-adapter architecture that bridges speech understanding and natural language reasoning. We employ a two-stage training pipeline: first, aligning semantic representations via an ASR objective; then, learning instruction-following abilities through synthetic audio-caption pairs automatically generated by prompting LLMs. Despite being trained on only 960 hours of ASR and 127 hours of synthetic instruction data, DIFFA demonstrates competitive performance on major benchmarks, including MMSU, MMAU, and VoiceBench, outperforming several autoregressive open-source baselines. Our results reveal the potential of diffusion-based language models for efficient and scalable audio understanding, opening a new direction for speech-driven AI. Our code will be available at https://github.com/NKU-HLT/DIFFA.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。