通过双模态对齐加速扩散模型语音合成训练，速度翻倍且效果更优。
Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment
- 利用文本与语音双重模态对齐，减少对扩散模型的依赖。
- 实验表明训练收敛速度提升一倍，性能优于基线方法。
- 适合关注语音合成效率优化的研究者与工程师。
本文旨在优化基于扩散模型的文语合成训练过程。尽管近期研究取得了显著进展，但其训练耗时长、计算成本高，主要源于扩散模型在学习复杂中间表示时的隐式引导。为此，我们提出A-DMA（双模态对齐加速训练）策略，引入一种新颖的对齐流程，同时利用文本和语音模态：文本引导对齐融合上下文表征，语音引导对齐精炼语义表示。通过将隐藏状态与判别性特征对齐，该训练方案降低了对扩散模型学习复杂表示的依赖。大量实验证明，A-DMA使收敛速度提升一倍，并在性能上超越基线方法。代码与演示样本见：https://github.com/ZhikangNiu/A-DMA。
原文摘要 · Abstract (English)
The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines. Code and demo samples are available at: https://github.com/ZhikangNiu/A-DMA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。