T5Gemma-TTS用编码器-解码器结构提升语音克隆质量，尤其在日语和韩语上表现突出。
T5Gemma-TTS Technical Report
- 采用编码器-解码器架构，通过跨注意力保持长语音中的文本控制力。
- 在日语上语音相似度达0.677，韩语达0.747，且字符错误率低至0.126。
- 引入进度感知旋转位置编码，显著提升语音时长控制，禁用后合成几乎失败。
自回归神经编解码语言模型虽具备强大零样本语音克隆能力，但解码器仅架构在输入文本作为前缀时会与音频序列争夺位置容量，导致长语音中文本条件弱化。本文提出T5Gemma-TTS，一种基于编码器-解码器的编解码语言模型，通过在每个解码层注入双向文本表示的交叉注意力，维持持续的文本条件。该模型基于预训练的T5Gemma编码器-解码器主干（2B编码器 + 2B解码器；共40亿参数），无需音素转换，直接处理子词级别文本。为提升时长控制，我们在全部26个交叉注意力层引入进度监控旋转位置嵌入（PM-RoPE），注入归一化进度信号以辅助解码器追踪目标语音长度。模型在17万小时多语言语音数据（英语、中文、日语）上训练。在日语上语音相似度达0.677，显著优于XTTSv2（0.622；95%置信区间无重叠）；韩语语音相似度达0.747（训练未包含韩语），虽高于XTTSv2（0.741）但差异不显著。日语字符错误率最低（0.126），尽管与Kokoro存在部分置信区间重叠。英语在LibriSpeech上的结果应视为上限估计，因LibriHeavy是其超集。使用相同检查点，禁用PM-RoPE后，字符错误率从0.129升至0.982，时长准确率从79%降至46%。代码与权重已公开于https://github.com/Aratako/T5Gemma-TTS。
原文摘要 · Abstract (English)
Autoregressive neural codec language models have shown strong zero-shot voice cloning ability, but decoder-only architectures treat input text as a prefix that competes with the growing audio sequence for positional capacity, weakening text conditioning over long utterances. We present T5Gemma-TTS, an encoder-decoder codec language model that maintains persistent text conditioning by routing bidirectional text representations through cross-attention at every decoder layer. Built on the T5Gemma pretrained encoder-decoder backbone (2B encoder + 2B decoder; 4B parameters), it inherits rich linguistic knowledge without phoneme conversion and processes text directly at the subword level. To improve duration control, we introduce Progress-Monitoring Rotary Position Embedding (PM-RoPE) in all 26 cross-attention layers, injecting normalized progress signals that help the decoder track target speech length. Trained on 170,000 hours of multilingual speech in English, Chinese, and Japanese, T5Gemma-TTS achieves a statistically significant speaker-similarity gain on Japanese over XTTSv2 (0.677 vs. 0.622; non-overlapping 95% confidence intervals) and the highest numerical Korean speaker similarity (0.747) despite Korean not being included in training, although this margin over XTTSv2 (0.741) is not statistically conclusive. It also attains the lowest numerical Japanese character error rate among five baselines (0.126), though this ranking should be interpreted cautiously because of partial confidence-interval overlap with Kokoro. English results on LibriSpeech should be viewed as an upper-bound estimate because LibriHeavy is a superset of LibriSpeech. Using the same checkpoint, disabling PM-RoPE at inference causes near-complete synthesis failure: CER degrades from 0.129 to 0.982 and duration accuracy drops from 79% to 46%. Code and weights are available at https://github.com/Aratako/T5Gemma-TTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。