首个支持长度控制与长语音生成的零样本语音合成模型
VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation
- 基于自回归编码解码器，引入进度监控旋转位置编码实现时长控制
- 在20-50秒长语音上显著提升自然度与可懂度，超越现有模型
- 适合需要灵活控制语音长度的语音合成应用
我们提出 VoiceStar，首个实现输出时长控制与外推能力的零样本语音合成模型。VoiceStar 是一个自回归编码解码神经编解码语言模型，采用新型进度监控旋转位置编码（PM-RoPE），并通过连续提示混合训练（CPM）进行训练。PM-RoPE 使模型更准确对齐文本与语音标记，指示目标生成时长，并支持生成比训练中见过更长的语音波形。CPM 训练缓解了训练与推理间的不匹配问题，显著提升了语音在说话人相似性和可懂度方面的质量。VoiceStar 在 LibriSpeech 与 Seed-TTS 等短语音基准上表现优于或持平当前最优模型，在 20-50 秒长语音/外推基准上，自然度与可懂度显著领先。代码与模型：https://github.com/jasonppy/VoiceStar。音频样本：https://jasonppy.github.io/VoiceStar_web
原文摘要 · Abstract (English)
We present VoiceStar, the first zero-shot TTS model that achieves both output duration control and extrapolation. VoiceStar is an autoregressive encoder-decoder neural codec language model, that leverages a novel Progress-Monitoring Rotary Position Embedding (PM-RoPE) and is trained with Continuation-Prompt Mixed (CPM) training. PM-RoPE enables the model to better align text and speech tokens, indicates the target duration for the generated speech, and also allows the model to generate speech waveforms much longer in duration than those seen during. CPM training also helps to mitigate the training/inference mismatch, and significantly improves the quality of the generated speech in terms of speaker similarity and intelligibility. VoiceStar outperforms or is on par with current state-of-the-art models on short-form benchmarks such as Librispeech and Seed-TTS, and significantly outperforms these models on long-form/extrapolation benchmarks (20-50s) in terms of intelligibility and naturalness. Code and models: https://github.com/jasonppy/VoiceStar. Audio samples: https://jasonppy.github.io/VoiceStar_web
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。