CLEAR用连续潜变量实现低延迟高质量语音合成,无需离散化音频。
CLEAR: Continuous Latent Autoregressive Modeling for High-quality and Low-latency Speech Synthesis
- 直接建模连续音频潜变量,避免离散化损失
- 在LibriSpeech上达1.88%词错误率,实时因子仅0.29
- 支持流式合成,首帧延迟仅96ms,适合实时应用
自回归(AR)语言模型已成为零样本文本到语音(TTS)合成的强大解决方案,能从几秒音频提示生成自然语音。然而,依赖离散音频标记的传统AR-TTS系统在标记化过程中面临有损压缩问题,需更长的离散标记序列才能捕获与连续表示相当的信息,导致推理延迟增加并复杂化自回归建模。为解决此问题,本文提出连续潜变量自回归模型(CLEAR),一种统一的零样本TTS框架,直接建模连续音频表示。具体而言,CLEAR引入带捷径连接的增强变分自编码器,实现高压缩比,将波形映射为紧凑的连续潜变量;同时设计轻量级基于MLP的修正流头,独立处理每个隐藏状态,联合训练于单阶段框架中以建模连续潜变量的概率分布。实验表明,所提零样本CLEAR TTS可实现高质量、低延迟语音合成。相比当前最优(SOTA)TTS模型,CLEAR在鲁棒性、说话人相似度和自然度方面表现相当,且具备更低的实时因子(RTF)。尤其在LibriSpeech测试干净数据集上,其达到SOTA结果,词错误率为1.88%,实时因子为0.29。此外,CLEAR支持流式语音合成,首帧延迟仅为96ms,同时保持高质量语音生成能力。
原文摘要 · Abstract (English)
Autoregressive (AR) language models have emerged as powerful solutions for zero-shot text-to-speech (TTS) synthesis, capable of generating natural speech from a few seconds of audio prompts. However, conventional AR-based TTS systems relying on discrete audio tokens face the challenge of lossy compression during tokenization, requiring longer discrete token sequences to capture the same information as continuous ones, which adds inference latency and complicates AR modeling. To address this challenge, this paper proposes the Continuous Latent Autoregressive model (CLEAR), a unified zero-shot TTS framework that directly models continuous audio representations. More specifically, CLEAR introduces an enhanced variational autoencoder with shortcut connections, which achieves a high compression ratio to map waveforms into compact continuous latents. A lightweight MLP-based rectified flow head that operates independently for each hidden state is presented to model the continuous latent probability distribution, and trained jointly with the AR model within a single-stage framework. Experiments show that the proposed zero-shot CLEAR TTS can synthesize high-quality speech with low latency. Compared to state-of-the-art (SOTA) TTS models, CLEAR delivers competitive performance in robustness, speaker similarity and naturalness, while offering a lower real-time factor (RTF). In particular, CLEAR achieves SOTA results on the LibriSpeech test-clean dataset, with a word error rate of 1.88\% and an RTF of 0.29. Moreover, CLEAR facilitates streaming speech synthesis with a first-frame delay of 96ms, while maintaining high-quality speech synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。