Llasa+让语音合成更快更流畅,无需牺牲质量。
Llasa+: Free Lunch for Accelerated and Streaming Llama-Based Speech Synthesis
- 引入多标记预测模块,单步生成多个语音标记
- 验证机制防止错误传播,实现1.48倍加速
- 支持流式输出,适配各类大模型语音系统
近年来,基于大语言模型的文本到语音(TTS)技术在自然度和灵活性上取得显著进展。然而,现有自回归结构及大型模型(如Llasa)仍面临推理延迟高和无法流式合成的问题。为此,我们提出Llasa+,一种基于Llasa的加速与流式语音合成模型。具体而言,通过在冻结主干网络后引入两个即插即用的多标记预测(MTP)模块,使模型能在单个自回归步骤中预测多个标记。为缓解不准确MTP带来的误差传播,设计了一种新颖的验证算法,利用冻结主干网络对生成标记进行校验,从而在不损失生成质量的前提下实现加速。此外,设计因果解码器,实现从标记流中重建语音。大量实验表明,即使仅在LibriTTS上训练,Llasa+仍实现1.48倍速度提升,且该MTP与验证框架可应用于任意基于大语言模型的TTS系统。代码与模型已公开于https://github.com/ASLP-lab/LLaSA_Plus。
原文摘要 · Abstract (English)
Recent progress in text-to-speech (TTS) has achieved impressive naturalness and flexibility, especially with the development of large language model (LLM)-based approaches. However, existing autoregressive (AR) structures and large-scale models, such as Llasa, still face significant challenges in inference latency and streaming synthesis. To deal with the limitations, we introduce Llasa+, an accelerated and streaming TTS model built on Llasa. Specifically, to accelerate the generation process, we introduce two plug-and-play Multi-Token Prediction (MTP) modules following the frozen backbone. These modules allow the model to predict multiple tokens in one AR step. Additionally, to mitigate potential error propagation caused by inaccurate MTP, we design a novel verification algorithm that leverages the frozen backbone to validate the generated tokens, thus allowing Llasa+ to achieve speedup without sacrificing generation quality. Furthermore, we design a causal decoder that enables streaming speech reconstruction from tokens. Extensive experiments show that Llasa+ achieves a 1.48X speedup without sacrificing generation quality, despite being trained only on LibriTTS. Moreover, the MTP-and-verification framework can be applied to accelerate any LLM-based model. All codes and models are publicly available at https://github.com/ASLP-lab/LLaSA_Plus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。