让离散扩散模型更快更准，突破生成速度与质量瓶颈
The Diffusion Duality
- 基于高斯扩散的启发，设计课程学习策略加速训练
- 通过离散一致性蒸馏，实现仅需几步即可完成生成
- 适合追求高效文本生成的开发者和研究者
统一状态的离散扩散模型因其自我修正能力，在快速文本生成方面具有潜力，但通常性能落后于自回归模型和掩码扩散模型。本文提出Duo方法，利用高斯扩散过程自然衍生出统一状态扩散这一关键洞察，将高斯扩散中的先进技巧迁移至离散场景。首先，引入由高斯过程指导的课程学习策略，使训练速度提升一倍，显著降低方差；在7个基准中的3个上，零样本困惑度超越自回归模型。其次，提出离散一致性蒸馏（Discrete Consistency Distillation），将连续空间的一致性蒸馏扩展到离散设置，使扩散语言模型的采样速度提升两个数量级，实现少步生成。项目主页提供代码、模型检查点及视频教程：http://s-sahoo.github.io/duo
原文摘要 · Abstract (English)
Uniform-state discrete diffusion models hold the promise of fast text generation due to their inherent ability to self-correct. However, they are typically outperformed by autoregressive models and masked diffusion models. In this work, we narrow this performance gap by leveraging a key insight: Uniform-state diffusion processes naturally emerge from an underlying Gaussian diffusion. Our method, Duo, transfers powerful techniques from Gaussian diffusion to improve both training and sampling. First, we introduce a curriculum learning strategy guided by the Gaussian process, doubling training speed by reducing variance. Models trained with curriculum learning surpass autoregressive models in zero-shot perplexity on 3 of 7 benchmarks. Second, we present Discrete Consistency Distillation, which adapts consistency distillation from the continuous to the discrete setting. This algorithm unlocks few-step generation in diffusion language models by accelerating sampling by two orders of magnitude. We provide the code, model checkpoints, and video tutorials on the project page: http://s-sahoo.github.io/duo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。