用声码器特征替代波形做对抗训练，大幅降低时序和内存开销。
Vocoder-Projected Feature Discriminator
- 用声码器提取特征进行对抗训练，避免直接处理波形
- 在扩散模型语音转换中达到与波形判别器相当的效果
- 训练时间减少9.6倍，内存消耗降低11.4倍，适合高效训练
在文本转语音（TTS）和语音转换（VC）中，常使用梅尔频谱等声学特征作为合成或转换目标，因其紧凑且易于学习。然而，最终目标是生成高质量波形，因此通过声码器将特征转换为波形，并在时域进行对抗训练是合理做法。但波形上采样带来显著的时间与内存开销。为此，我们提出声码器投影特征判别器（VPFD），利用声码器特征进行对抗训练。在基于扩散模型的语音转换蒸馏实验中，仅需一个上采样步骤的预训练冻结声码器特征提取器，即可实现与波形判别器相当的语音转换性能，同时将训练时间减少9.6倍，内存消耗降低11.4倍。
原文摘要 · Abstract (English)
In text-to-speech (TTS) and voice conversion (VC), acoustic features, such as mel spectrograms, are typically used as synthesis or conversion targets owing to their compactness and ease of learning. However, because the ultimate goal is to generate high-quality waveforms, employing a vocoder to convert these features into waveforms and applying adversarial training in the time domain is reasonable. Nevertheless, upsampling the waveform introduces significant time and memory overheads. To address this issue, we propose a vocoder-projected feature discriminator (VPFD), which uses vocoder features for adversarial training. Experiments on diffusion-based VC distillation demonstrated that a pretrained and frozen vocoder feature extractor with a single upsampling step is necessary and sufficient to achieve a VC performance comparable to that of waveform discriminators while reducing the training time and memory consumption by 9.6 and 11.4 times, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。