用文本和声音信息生成自然语调，提升语音合成真实感。
ProMode: A Speech Prosody Model Conditioned on Acoustic and Textual Inputs
- 输入文本和声学特征，生成固定长度的语调嵌入向量。
- 在不同粒度下预测音高和能量均优于当前最佳方法。
- 可直接用于语音合成，听感更自然，适合需要语调建模的任务。
语调承载了语音中丰富的情感与语义信息以及个体差异。我们提出一个独立的语音语调模型，可将文本映射为音高（F0）和能量等语调特征，并应用于语音合成（TTS）等下游任务。ProMode 编码器输入经过部分掩码的声学特征与对齐的文本内容，生成固定长度的语调嵌入。解码器利用编码后的语调信息和未掩码的文本内容，在掩码区域预测声学特征。在 GigaSpeech 数据集上训练后，与现有先进风格编码器对比，本模型在不同粒度下对 F0 与能量的预测均取得持续改进。将预测的语调特征集成至 TTS 系统后进行感知测试，结果表明其语调表现更受青睐，验证了该模型在语调建模关键任务中的潜力。
原文摘要 · Abstract (English)
Prosody conveys rich emotional and semantic information of the speech signal as well as individual idiosyncrasies. We propose a stand-alone model that maps text-to-prosodic features such as F0 and energy and can be used in downstream tasks such as TTS. The ProMode encoder takes as input acoustic features and time-aligned textual content, both are partially masked, and obtains a fixed-length latent prosodic embedding. The decoder predicts acoustics in the masked region using both the encoded prosody input and unmasked textual content. Trained on the GigaSpeech dataset, we compare our method with state-of-the-art style encoders. For F0 and energy predictions, we show consistent improvements for our model at different levels of granularity. We also integrate these predicted prosodic features into a TTS system and conduct perceptual tests, which show higher prosody preference compared to the baselines, demonstrating the model's potential in tasks where prosody modeling is important.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。