通过并行结构融合语义与声学信息，提升零样本语音合成质量
Parallel GPT: Harmonizing the Independence and Interdependence of Acoustic and Semantic Information for Zero-Shot Text-to-Speech
- 采用自回归与非自回归模块协同，同步生成语义与声学标记
- 在英、中文数据集上显著优于现有零样本语音合成模型
- 适合需要高质量、高效语音合成的开发者与研究者
语音表征与大语言模型的发展提升了零样本文本到语音（TTS）的性能。然而，现有零样本TTS模型难以捕捉声学与语义特征间的复杂关联，导致表现力与相似性不足。根本原因在于语义与声学特征同时具备独立性与相互依赖性。本文提出Parallel GPT框架，结合自回归（AR）与非自回归（NAR）模块，以协调二者关系。AR模型利用提出的并行分词器，同步生成顶层语义与声学标记；NAR模型则基于AR输出，预测细节标记以体现依赖关系。该架构支持并行生成，提升效率。在英文与中文数据集上的实验表明，所提模型在音质与合成效率上均显著优于现有方法。语音样例可访问 https://t1235-ch.github.io/pgpt/。
原文摘要 · Abstract (English)
Advances in speech representation and large language models have enhanced zero-shot text-to-speech (TTS) performance. However, existing zero-shot TTS models face challenges in capturing the complex correlations between acoustic and semantic features, resulting in a lack of expressiveness and similarity. The primary reason lies in the complex relationship between semantic and acoustic features, which manifests independent and interdependent aspects.This paper introduces a TTS framework that combines both autoregressive (AR) and non-autoregressive (NAR) modules to harmonize the independence and interdependence of acoustic and semantic information. The AR model leverages the proposed Parallel Tokenizer to synthesize the top semantic and acoustic tokens simultaneously. In contrast, considering the interdependence, the Coupled NAR model predicts detailed tokens based on the general AR model's output. Parallel GPT, built on this architecture, is designed to improve zero-shot text-to-speech synthesis through its parallel structure. Experiments on English and Chinese datasets demonstrate that the proposed model significantly outperforms the quality and efficiency of the synthesis of existing zero-shot TTS models. Speech demos are available at https://t1235-ch.github.io/pgpt/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。