分离音色与语调控制，让语音合成更灵活可控。
DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance
- 用多模态提示分离音色与语调，实现独立调节。
- 在 LJSpeech 与 VCTK 数据集上，风格控制力优于开源基线。
- 适合需要精细语音风格调控的研究与应用者。
可控文本转语音系统在独立操控说话人音色与语调方面面临挑战，常出现属性纠缠问题。本文提出 DMP-TTS，一种基于潜在扩散变换器（DiT）的框架，具备显式解耦与多模态提示机制。通过基于 CLAP 的风格编码器（Style-CLAP），将参考音频与描述性文本的线索对齐至共享空间，并采用对比学习与多任务监督训练风格属性。推理时引入分层条件丢弃训练的链式无分类器引导（cCFG），实现内容、音色与风格引导强度的独立调节。此外，采用表示对齐（REPA）技术，从预训练 Whisper 模型中蒸馏声学-语义特征注入中间 DiT 表示，稳定训练并加速收敛。实验表明，DMP-TTS 在保持良好可懂度与自然度的同时，显著提升风格可控性，优于现有开源基线。代码与演示将发布于 https://y61329697.github.io/DMP-TTS/。
原文摘要 · Abstract (English)
Controllable text-to-speech (TTS) systems face significant challenges in achieving independent manipulation of speaker timbre and speaking style, often suffering from entanglement between these attributes. We present DMP-TTS, a latent Diffusion Transformer (DiT) framework with explicit disentanglement and multi-modal prompting. A CLAP-based style encoder (Style-CLAP) aligns cues from reference audio and descriptive text in a shared space and is trained with contrastive learning plus multi-task supervision on style attributes. For fine-grained control during inference, we introduce chained classifier-free guidance (cCFG) trained with hierarchical condition dropout, enabling independent adjustment of content, timbre, and style guidance strengths. Additionally, we employ Representation Alignment (REPA) to distill acoustic-semantic features from a pretrained Whisper model into intermediate DiT representations, stabilizing training and accelerating convergence. Experiments show that DMP-TTS delivers stronger style controllability than open-source baselines while maintaining competitive intelligibility and naturalness. Code and demos will be available at https://y61329697.github.io/DMP-TTS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。