arXiv:2410.11097eess.AScs.AI2024-10ICML被引 7

用蒸馏扩散模型直接优化语音合成指标,速度更快、质量更高。

DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis

  • 通过可微分路径实现端到端优化,打通所有模块的梯度
  • 相比教师模型,推理速度提升数量级,自然度和相似度显著提高
  • 适合关注语音质量与效率的生成模型研究者

扩散模型在语音合成(包括文本到语音TTS和语音克隆)中展现出巨大潜力,但其迭代去噪过程计算开销大,现有蒸馏方法普遍存在质量下降问题。此外,传统TTS方法受限于不可微组件或迭代采样,无法实现感知指标的端到端优化。本文提出DMOSpeech,一种基于蒸馏扩散模型的TTS系统,首次实现所有模块的可微分优化,支持连接时序分类(CTC)损失和说话人验证(SV)损失的直接梯度传播。大量实验及人工评估表明,该方法在保持高自然度、可懂度和说话人相似度的同时,推理时间减少数个数量级。本工作建立了一种通过直接优化感知指标对齐人类听觉偏好的新范式。音频样本见 https://dmospeech.github.io/。

原文摘要 · Abstract (English)

Diffusion models have demonstrated significant potential in speech synthesis tasks, including text-to-speech (TTS) and voice cloning. However, their iterative denoising processes are computationally intensive, and previous distillation attempts have shown consistent quality degradation. Moreover, existing TTS approaches are limited by non-differentiable components or iterative sampling that prevent true end-to-end optimization with perceptual metrics. We introduce DMOSpeech, a distilled diffusion-based TTS model that uniquely achieves both faster inference and superior performance compared to its teacher model. By enabling direct gradient pathways to all model components, we demonstrate the first successful end-to-end optimization of differentiable metrics in TTS, incorporating Connectionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss. Our comprehensive experiments, validated through extensive human evaluation, show significant improvements in naturalness, intelligibility, and speaker similarity while reducing inference time by orders of magnitude. This work establishes a new framework for aligning speech synthesis with human auditory preferences through direct metric optimization. The audio samples are available at https://dmospeech.github.io/.

语音合成扩散模型端到端可微分

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。