arXiv:2509.22062cs.SDeess.AS2025-09

用双阶段模型让语音合成更准更稳,零样本也能生成自然语音。

Comprehend and Talk: Text to Speech Synthesis via Dual Language Modeling

  • 分层语音编码器注入语义信息,减少学习负担。
  • 先理解后生成,跨模态对齐提升语音语义一致性。
  • 动态纠错机制显著提升生成稳定性,适合高要求语音任务。

基于大语言模型的自回归语音合成系统虽已达顶尖质量,但仍面临挑战。其核心是通过神经音频编解码器将连续语音波形离散化为离散标记序列,但单一码书建模易造成信息损失;通常由残差向量量化(RVQ)生成的分层声学标记缺乏明确语义结构,给模型带来沉重学习负担。此外,自回归过程固有的误差累积会降低生成稳定性。为此,我们提出CaT-TTS,一种鲁棒且语义引导的零样本语音合成新框架。首先,引入S3Codec,一种分裂式RVQ编解码器,通过从先进语音识别模型中进行语义蒸馏,将显式语言特征注入主码书,提供结构化表示以简化学习任务。其次,提出“理解-生成”双Transformer架构,解耦理解与渲染过程:初始“理解”变压器建模文本与音频语义标记间的跨模态关系,形成高层话语计划;后续“生成”变压器则依此计划自回归合成分层声学标记。最后,为增强生成稳定性,提出掩码音频并行推理(MAPI),一种近乎无参数的推理策略,可动态引导解码过程以缓解局部错误。

原文摘要 · Abstract (English)

Existing Large Language Model (LLM) based autoregressive (AR) text-to-speech (TTS) systems, while achieving state-of-the-art quality, still face critical challenges. The foundation of this LLM-based paradigm is the discretization of the continuous speech waveform into a sequence of discrete tokens by neural audio codec. However, single codebook modeling is well suited to text LLMs, but suffers from significant information loss; hierarchical acoustic tokens, typically generated via Residual Vector Quantization (RVQ), often lack explicit semantic structure, placing a heavy learning burden on the model. Furthermore, the autoregressive process is inherently susceptible to error accumulation, which can degrade generation stability. To address these limitations, we propose CaT-TTS, a novel framework for robust and semantically-grounded zero-shot synthesis. First, we introduce S3Codec, a split RVQ codec that injects explicit linguistic features into its primary codebook via semantic distillation from a state-of-the-art ASR model, providing a structured representation that simplifies the learning task. Second, we propose an ``Understand-then-Generate'' dual-Transformer architecture that decouples comprehension from rendering. An initial ``Understanding'' Transformer models the cross-modal relationship between text and the audio's semantic tokens to form a high-level utterance plan. A subsequent ``Generation'' Transformer then executes this plan, autoregressively synthesizing hierarchical acoustic tokens. Finally, to enhance generation stability, we introduce Masked Audio Parallel Inference (MAPI), a nearly parameter-free inference strategy that dynamically guides the decoding process to mitigate local errors.

语音合成大模型双阶段语义建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。