arXiv:2504.12339cs.CLcs.SD2025-04被引 3

用双分支架构让大模型生成更自然的语音,解决音色丢失和语义遗忘问题。

GOAT-TTS: Expressive and Realistic Speech Generation via A Dual-Branch LLM

  • 双分支设计:一边对齐语音与文本,一边保留语言理解能力
  • 在真实语音数据上表现接近顶尖模型,支持方言语音合成
  • 适合需要高保真语音生成和实时输出的场景

尽管大语言模型通过离散标记化范式革新了文语转换技术,但现有架构在三个关键维度上存在根本性矛盾:1)语音提示量化导致声学特征不可逆丢失;2)对精确对齐的语音-文本配对高度依赖,限制实际部署;3)优化语音标记生成时导致模型原有文本理解能力严重遗忘。为此,我们提出基于大语言模型的语音生成框架——通过新颖双分支架构优化(GOAT-TTS)。该框架引入两项核心创新:(1) 模态对齐分支结合语音编码器与投影模块,捕获连续声学嵌入,实现韵律特征(语言、音色、情感)与语义文本表示间的双向关联,无需依赖转录文本;(2) 语音生成分支在大语言模型顶层进行模块化微调以预测语音标记,同时冻结底层n层以保持基础语言知识。此外,引入多标记预测机制,支持实时流式语音合成。实验表明,所提方法性能可媲美当前最优文语转换模型,并验证了合成方言语音数据的有效性。

原文摘要 · Abstract (English)

While large language models (LLMs) have revolutionized text-to-speech (TTS) synthesis through discrete tokenization paradigms, current architectures exhibit fundamental tensions between three critical dimensions: 1) irreversible loss of acoustic characteristics caused by quantization of speech prompts; 2) stringent dependence on precisely aligned prompt speech-text pairs that limit real-world deployment; and 3) catastrophic forgetting of the LLM's native text comprehension during optimization for speech token generation. To address these challenges, we propose an LLM-based text-to-speech Generation approach Optimized via a novel dual-branch ArchiTecture (GOAT-TTS). Our framework introduces two key innovations: (1) The modality-alignment branch combines a speech encoder and projector to capture continuous acoustic embeddings, enabling bidirectional correlation between paralinguistic features (language, timbre, emotion) and semantic text representations without transcript dependency; (2) The speech-generation branch employs modular fine-tuning on top-k layers of an LLM for speech token prediction while freezing the bottom-n layers to preserve foundational linguistic knowledge. Moreover, multi-token prediction is introduced to support real-time streaming TTS synthesis. Experimental results demonstrate that our GOAT-TTS achieves performance comparable to state-of-the-art TTS models while validating the efficacy of synthesized dialect speech data.

语音生成大模型双分支实时合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。