arXiv:2509.20072cs.CL2025-09被引 5

提出统一框架,让文本和语音并行生成,提升对话系统效率。

From Text to Talk: Audio-Language Model Needs Non-Autoregressive Joint Training

  • 用非自回归扩散模型合成语音,结合自回归文本生成,统一训练。
  • 在多个语音任务上超越现有自回归与非自回归方法,提升显著。
  • 适合研究多模态生成、语音合成及高效对话系统的开发者。

近期大语言模型的发展推动了多模态应用,尤其在语音对话系统中。现有模型多采用自回归方法处理交错的音频与文本,但忽视了文本依赖目标间关系,而音频主要依赖源-目标关系。本文提出 Text-to-Talk(TtT)框架,将自回归文本生成与非自回归音频扩散统一于单一 Transformer 模型中。通过利用吸收离散扩散的任意顺序自回归特性,实现文本与音频的统一训练目标。设计模态感知注意力机制,对文本强制因果解码,允许音频内部双向建模,并引入三种训练策略以减少训练与测试差异。推理时,采用块级扩散并行生成音频,灵活处理可变长度输出。在 Audio-QA、ASR、AAC 及语音对话基准上,TtT 均显著优于强基线,消融实验验证各组件有效性。代码、数据与模型将开源,促进该方向研究。

原文摘要 · Abstract (English)

Recent advances in large language models (LLMs) have attracted significant interest in extending their capabilities to multimodal scenarios, particularly for speech-to-speech conversational systems. However, existing multimodal models handling interleaved audio and text rely on autoregressive (AR) methods, overlooking that text depends on target-target relations whereas audio depends mainly on source-target relations. In this work, we propose Text-to-Talk (TtT), a unified audio-text framework that integrates AR text generation with non-autoregressive (NAR) audio diffusion in a single Transformer. By leveraging the any-order AR property of absorbing discrete diffusion, our approach provides a unified training objective for text and audio. To support this hybrid generation paradigm, we design a modality-aware attention mechanism that enforces causal decoding for text while allowing bidirectional modeling within audio spans, and further introduce three training strategies that reduce train-test discrepancies. During inference, TtT employs block-wise diffusion to synthesize audio in parallel while flexibly handling variable-length outputs. Comprehensive experiments on Audio-QA, ASR, AAC and speech-to-speech benchmarks show that TtT consistently surpasses strong AR and NAR baselines, with additional ablation and training-strategy analyses confirming the contribution of each component. We will open-source our models, data and code to facilitate future research in this direction.

语音生成多模态扩散模型非自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。