提出新型采样器，让离散扩散模型生成更优且步数越多越好。
The Diffusion Duality, Chapter II: $Ψ$-Samplers
- 设计基于预测-修正的新型采样器，适用于任意噪声过程。
- 在语言和图像任务上优于传统采样，步数增加仍持续提升质量。
- 适合关注高效生成与训练优化的研究者或工程实践者。
均匀状态离散扩散模型因其自校正能力，在少步生成与引导任务中表现优异，优于自回归或掩码扩散模型。然而，使用传统祖先采样器时，其生成质量随步数增加趋于饱和。本文提出一类通用的预测-修正（PC）采样器，适用于任意噪声过程。与均匀状态扩散模型结合后，该方法在语言与图像建模任务中均超越祖先采样：在OpenWebText上达到更低的生成困惑度（相同一元熵下），在CIFAR10上获得更优的FID与IS分数。关键优势在于，不同于传统采样器，本方法随采样步数增加仍持续改进。这挑战了掩码扩散是扩散语言建模必然方向的假设。此外，本文还设计了一种内存高效的高斯松弛训练课程，相较Duo方法，训练时间减少25%，内存降低33%，同时保持OpenWebText与LM1B上的相近困惑度及强下游性能。代码、模型权重与视频教程已公开：https://s-sahoo.com/duo-ch2
原文摘要 · Abstract (English)
Uniform-state discrete diffusion models excel at few-step generation and guidance due to their ability to self-correct, making them preferred over autoregressive or Masked diffusion models in these settings. However, their sampling quality plateaus with ancestral samplers as the number of steps increases. We introduce a family of Predictor-Corrector (PC) samplers for discrete diffusion that generalize prior methods and apply to arbitrary noise processes. When paired with uniform-state diffusion, our samplers outperform ancestral sampling on both language and image modeling, achieving lower generative perplexity at matched unigram entropy on OpenWebText and better FID/IS scores on CIFAR10. Crucially, unlike conventional samplers, our PC methods continue to improve with more sampling steps. Taken together, these findings call into question the assumption that Masked diffusion is the inevitable future of diffusion-based language modeling. Beyond sampling, we develop a memory-efficient curriculum for the Gaussian relaxation training phase, reducing training time by 25% and memory by 33% compared to Duo while maintaining comparable perplexity on OpenWebText and LM1B and strong downstream performance. We release code, checkpoints, and a video-tutorial on: https://s-sahoo.com/duo-ch2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。