融合自回归与流匹配，实现高效高质语音合成
Next Tokens Denoising for Speech Synthesis
- 分块处理48kHz音频编码令牌，每秒12.5个，支持跨块自回归建模
- 在块内并行流匹配加速去噪，生成速度显著提升
- 适合长语音如播客生成，零样本生成质量高
尽管扩散模型和自回归（AR）模型在生成建模方面取得显著进展，但各自存在局限：AR模型依赖因果注意力，无法利用未来上下文，生成速度慢；扩散模型则面临键值缓存难题。为此，我们提出Dragon-FM，一种新型文本到语音（TTS）架构，统一AR与流匹配机制。该模型以12.5个令牌/秒的紧凑速率处理48 kHz音频编码令牌，支持跨块自回归建模以保证全局连贯性，同时在块内实现并行流匹配，加速迭代去噪。该设计使模型能跨块使用键值缓存，并在块内利用双向上下文。此外，它打通了连续与离散特征建模，证明连续的自回归流匹配可借助有限标量量化预测离散令牌。这一高效的编码方案与快速的分块自回归结构，使其在生成长语音内容（如播客）方面表现优异。在播客数据集上的实验表明，该模型能高效生成高质量的零样本播客。
原文摘要 · Abstract (English)
While diffusion and autoregressive (AR) models have significantly advanced generative modeling, they each present distinct limitations. AR models, which rely on causal attention, cannot exploit future context and suffer from slow generation speeds. Conversely, diffusion models struggle with key-value (KV) caching. To overcome these challenges, we introduce Dragon-FM, a novel text-to-speech (TTS) design that unifies AR and flow-matching. This model processes 48 kHz audio codec tokens in chunks at a compact rate of 12.5 tokens per second. This design enables AR modeling across chunks, ensuring global coherence, while parallel flow-matching within chunks facilitates fast iterative denoising. Thus, the model leverages KV-cache across chunks and utilizes bidirectional context within each chunk. Furthermore, it bridges continuous and discrete feature modeling, demonstrating that continuous AR flow-matching can predict discrete tokens with finite scalar quantizers. This efficient codec and fast chunk-autoregressive architecture also make the model highly effective for generating long-form content, such as podcasts. Experiments on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。