多语言语音合成模型通过偏好优化提升跨语言发音人一致性。
LatinX: Aligning a Multilingual TTS Model with Direct Preference Optimization
- 三阶段训练：文本到音频预训练、零样本克隆微调、基于WER与相似度的偏好优化。
- 相比基线，WER降低且发音人相似度客观指标提升，主观评估更优。
- 适合语音克隆、跨语言语音翻译场景，关注发音人身份保持的研究者可重点关注。
我们提出 LatinX，一个用于级联式语音到语音翻译的多语言文本转语音模型，能跨语言保留源发音人的身份特征。LatinX 是一个 12 层的解码器仅 Transformer 模型，分三个阶段训练：(i) 文本到音频映射的预训练；(ii) 零样本语音克隆的监督微调；(iii) 基于词错误率（WER）和说话人相似度指标自动标注的偏好对，使用直接偏好优化（DPO）进行对齐。模型在英语和罗曼语系语言（重点为葡萄牙语）上训练，加入 DPO 后，WER 持续降低，客观相似度指标优于微调基线。人工评估进一步显示，其感知发音人相似度高于强基线模型 XTTSv2，揭示了客观与主观度量间的差距。我们提供了跨语言分析，并讨论未来工作方向，包括平衡偏好信号和低延迟架构。
原文摘要 · Abstract (English)
We present LatinX, a multilingual text-to-speech (TTS) model for cascaded speech-to-speech translation that preserves the source speaker's identity across languages. LatinX is a 12-layer decoder-only Transformer trained in three stages: (i) pre-training for text-to-audio mapping, (ii) supervised fine-tuning for zero-shot voice cloning, and (iii) alignment with Direct Preference Optimization (DPO) using automatically labeled pairs based on Word Error Rate (WER) and speaker-similarity metrics. Trained on English and Romance languages with emphasis on Portuguese, LatinX with DPO consistently reduces WER and improves objective similarity over the fine-tuned baseline. Human evaluations further indicate stronger perceived speaker similarity than a strong baseline (XTTSv2), revealing gaps between objective and subjective measures. We provide cross-lingual analyses and discuss balanced preference signals and lower-latency architectures as future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。