arXiv:2510.12210eess.AScs.CL2025-10被引 4

用离散向量量化空间实现语音生成,兼具高效与可控性。

DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation

  • 用自回归模型草拟分块离散码,再并行扩散填充完成语音合成。
  • 在零样本条件下超越现有系统,在自然度和一致性上提升显著。
  • 支持推理时动态调整比特率与计算量,适合实时或低资源场景。

近期将自回归草图模型与基于扩散的精修模型结合的方法虽有潜力,但在分布偏移下仍脆弱且可控性有限。我们提出DiSTAR,一种完全在离散残差向量量化(RVQ)码空间运行的零样本文语转换框架,将自回归语言模型与掩码扩散模型紧密耦合,无需强制对齐或持续时间预测器。具体而言,DiSTAR利用自回归模型草拟块级RVQ码,随后基于草图进行并行掩码扩散填充以完成下一区块,实现长序列合成的同时保持块级并行性,并缓解经典自回归暴露偏差。离散码空间提供显式控制:在推理阶段,DiSTAR采用贪婪与采样解码方式均能生成高质量音频,支持分类器自由引导,可调节鲁棒性与多样性之间的权衡,并通过测试时对RVQ层的剪枝实现可变比特率与可控计算量。大量实验与消融分析表明,DiSTAR在鲁棒性、自然度及说话人/风格一致性方面优于现有最先进零样本TTS系统,同时保持丰富的输出多样性。音频样例见 https://anonymous.4open.science/w/DiSTAR_demo。

原文摘要 · Abstract (English)

Recent attempts to interleave autoregressive (AR) sketchers with diffusion-based refiners over continuous speech representations have shown promise, but they remain brittle under distribution shift and offer limited levers for controllability. We introduce DISTAR, a zero-shot text-to-speech framework that operates entirely in a discrete residual vector quantization (RVQ) code space and tightly couples an AR language model with a masked diffusion model, without forced alignment or a duration predictor. Concretely, DISTAR drafts block-level RVQ tokens with an AR language model and then performs parallel masked-diffusion infilling conditioned on the draft to complete the next block, yielding long-form synthesis with blockwise parallelism while mitigating classic AR exposure bias. The discrete code space affords explicit control at inference: DISTAR produces high-quality audio under both greedy and sample-based decoding using classifier-free guidance, supports trade-offs between robustness and diversity, and enables variable bit-rate and controllable computation via RVQ layer pruning at test time. Extensive experiments and ablations demonstrate that DISTAR surpasses state-of-the-art zero-shot TTS systems in robustness, naturalness, and speaker/style consistency, while maintaining rich output diversity. Audio samples are provided on https://anonymous.4open.science/w/DiSTAR_demo.

语音生成扩散模型离散表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。