让语音合成像大模型一样同时扩大训练和推理计算,提升自然度与情感表现。
Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis
- 用单一Transformer架构和向量量化编码器,统一语音合成流程。
- 训练算力越大,语音越自然,韵律越复杂;推理算力越大,情感越丰富、音色越一致。
- 适合追求高保真语音生成的研究者和开发者使用。
近期基于文本的大语言模型(如GPT系列和o1模型)表明,同时扩展训练和推理计算能显著提升性能。然而,当前基于大模型的语音合成(TTS)系统多为多阶段设计,需分别使用不同模型(如在大模型后接扩散模型),难以决定是否在训练或推理时扩展特定模型的算力。本文提出Llasa框架,通过单层向量量化(VQ)编码器和单一Transformer结构,与标准LLM(如Llama)完全对齐。实验表明,扩大训练算力可持续提升语音自然度,并生成更复杂的韵律模式;从推理算力扩展角度看,引入语音理解模型作为搜索过程中的验证器,发现增加推理算力会使采样模式偏向特定验证器偏好,从而提升情感表达力、音色一致性及内容准确性。此外,本文公开了1B、3B、8B规模的TTS模型及编码器的检查点与训练代码。
原文摘要 · Abstract (English)
Recent advances in text-based large language models (LLMs), particularly in the GPT series and the o1 model, have demonstrated the effectiveness of scaling both training-time and inference-time compute. However, current state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring separate models (e.g., diffusion models after LLM), complicating the decision of whether to scale a particular model during training or testing. This work makes the following contributions: First, we explore the scaling of train-time and inference-time compute for speech synthesis. Second, we propose a simple framework Llasa for speech synthesis that employs a single-layer vector quantizer (VQ) codec and a single Transformer architecture to fully align with standard LLMs such as Llama. Our experiments reveal that scaling train-time compute for Llasa consistently improves the naturalness of synthesized speech and enables the generation of more complex and accurate prosody patterns. Furthermore, from the perspective of scaling inference-time compute, we employ speech understanding models as verifiers during the search, finding that scaling inference-time compute shifts the sampling modes toward the preferences of specific verifiers, thereby improving emotional expressiveness, timbre consistency, and content accuracy. In addition, we released the checkpoint and training code for our TTS model (1B, 3B, 8B) and codec model publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。