通过偏好对齐提升零样本语音合成在复杂场景下的可懂度
Advancing Zero-shot Text-to-Speech Intelligibility across Diverse Domains via Preference Alignment
- 用偏好对齐构建跨域训练数据，增强模型泛化能力
- 新数据集INTP使多模型在绕口令等场景下可懂度显著提升
- 适合需要高鲁棒性语音合成的工业应用与研究者
现代零样本文本到语音（TTS）系统虽经大规模预训练，但在绕口令、重复词、代码切换和跨语言合成等挑战性场景中仍存在可懂度问题。本文提出利用偏好对齐技术，针对性构建超出预训练分布的数据以提升性能。我们引入新数据集Intelligibility Preference Speech Dataset（INTP），并将直接偏好优化（DPO）框架扩展至多种TTS架构。经过INTP对齐后，多个TTS模型在多样领域中不仅可懂度提升，自然度、相似度和音频质量也整体改善。进一步验证了INTP在更优模型如CosyVoice 2和Ints上的弱到强泛化能力，并展示了基于Ints的迭代对齐可实现持续优化。音频样本见https://intalign.github.io/。
原文摘要 · Abstract (English)
Modern zero-shot text-to-speech (TTS) systems, despite using extensive pre-training, often struggle in challenging scenarios such as tongue twisters, repeated words, code-switching, and cross-lingual synthesis, leading to intelligibility issues. To address these limitations, this paper leverages preference alignment techniques, which enable targeted construction of out-of-pretraining-distribution data to enhance performance. We introduce a new dataset, named the Intelligibility Preference Speech Dataset (INTP), and extend the Direct Preference Optimization (DPO) framework to accommodate diverse TTS architectures. After INTP alignment, in addition to intelligibility, we observe overall improvements including naturalness, similarity, and audio quality for multiple TTS models across diverse domains. Based on that, we also verify the weak-to-strong generalization ability of INTP for more intelligible models such as CosyVoice 2 and Ints. Moreover, we showcase the potential for further improvements through iterative alignment based on Ints. Audio samples are available at https://intalign.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。