用对比学习让语音合成理解文本真实语义,兼顾音质与速度。
Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding
- 用Clip框架连接文本与真实频谱图,增强语义理解。
- 在LJSpeech和Baker数据集上达到顶尖主观评分(MOS)。
- 基于Transformer结构,推理速度快,适合实际应用。
传统语音合成方法主要建立音素与梅尔频谱图之间的映射关系,但在音素编码阶段缺乏真实频谱图的辅助信息,导致编码过程缺乏真正的语义理解。同时,现有系统难以平衡模型推理速度与合成语音质量:高质量方法通常速度慢,快速方法则牺牲音质。本文提出Clip-TTS,基于Clip架构的文本到语音合成方法。该方法在文本编码阶段引入真实梅尔频谱图作为对比信号,使文本编码器直接学习全局上下文的真实语义,从而保障语音合成质量。模型采用基础Transformer结构,实现快速推理。实验表明,Clip-TTS在LJSpeech和Baker数据集上达到最先进的主观评分(MOS),并在多情感数据集上表现优异。音频样例可访问:https://ltydd1314.github.io/。
原文摘要 · Abstract (English)
Traditional text-to-speech (TTS) methods primarily focus on establishing a mapping between phonemes and mel-spectrograms. However, during the phoneme encoding stage, there is often a lack of real mel-spectrogram auxiliary information, which results in the encoding process lacking true semantic understanding. At the same time, traditional TTS systems often struggle to balance the inference speed of the model with the quality of the synthesized speech. Methods that generate high-quality synthesized speech tend to have slower inference speeds, while faster inference methods often sacrifice speech quality. In this paper, I propose Clip-TTS, a TTS method based on the Clip architecture. This method uses the Clip framework to establish a connection between text content and real mel-spectrograms during the text encoding stage, enabling the text encoder to directly learn the true semantics of the global context, thereby ensuring the quality of the synthesized speech. In terms of model architecture, I adopt the basic structure of Transformer, which allows Clip-TTS to achieve fast inference speeds. Experimental results show that on the LJSpeech and Baker datasets, the speech generated by Clip-TTS achieves state-of-the-art MOS scores, and it also performs excellently on multi-emotion datasets.Audio samples are available at: https://ltydd1314.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。