让语音模型先思考再说话，提升回答准确性和语音质量。
DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion
- 用统一扩散模型同步生成思考文本和语音输出。
- 语音问答准确率领先9个百分点，语音合成错误率仅6.2%。
- 适合需要高精度语音交互的智能助手场景。
现有语音语言模型直接生成回答，缺乏可修正的推理过程。本文提出「无声思考，有声回应」新范式，让语音大模型在生成语音前先输出文本推理过程，用思考内容优化语音质量。为此，我们提出 extbf{DiffuSpeech}，首个基于扩散模型的语音-文本联合建模框架，将离散文本与分词语音统一在掩码扩散机制下。不同于自回归方法，该模型通过迭代去噪同步生成推理痕迹与语音标记，采用模态特异性掩码策略。我们构建了首个包含配对文本推理痕迹的语音问答数据集 extbf{ extit{SpeechQA-TT}}，含26,000个样本，总计319小时。实验表明， extbf{DiffuSpeech} 在语音到语音问答任务中达到最先进水平，最高优于基线9个百分点；在生成式模型中实现最佳语音合成质量（6.2% WER），同时保持良好语言理解能力（66.2% MMLU）。消融实验验证了扩散架构与思考痕迹均带来性能提升。
原文摘要 · Abstract (English)
Current speech language models generate responses directly without explicit reasoning, leading to errors that cannot be corrected once audio is produced. We introduce \textbf{``Silent Thought, Spoken Answer''} -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses, with thinking traces informing speech quality. To realize this, we present \method{}, the first diffusion-based speech-text language model supporting both understanding and generation, unifying discrete text and tokenized speech under a single masked diffusion framework. Unlike autoregressive approaches, \method{} jointly generates reasoning traces and speech tokens through iterative denoising, with modality-specific masking schedules. We also construct \dataset{}, the first speech QA dataset with paired text reasoning traces, containing 26K samples totaling 319 hours. Experiments show \method{} achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points, while attaining the best TTS quality among generative models (6.2\% WER) and preserving language understanding (66.2\% MMLU). Ablations confirm that both the diffusion architecture and thinking traces contribute to these gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。