高斯扩散模型在离散数据上失效，因采样时易落入低密度区。
Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?
- 发现采样过程中的关键区间导致数据密度多峰化
- 引入自条件与q采样可显著提升生成质量
- 适用于文本、代码、蛋白质等离散生成任务
扩散模型在连续空间中已成为生成建模的标准方法，但在离散数据上的应用仍具挑战。本文研究发现，以DDPM求解器为基础的高斯扩散模型在将离散分布表示为连续空间中的狄拉克混合分布时，会在特定采样区间内出现数据密度多峰化现象。在此区间，DDPM偶尔会进入模式间的低密度区域，产生分布外输入，从而降低生成样本质量。我们验证了现有启发式方法（如自条件化和一种称为q采样）可缓解该问题。进一步实验表明，在关键区间内结合自条件化与从DDPM切换至q采样，能显著提升真实数据上的生成质量。该结论在多个领域（包括文本、编程代码、蛋白质）的条件与无条件任务中均得到验证。
原文摘要 · Abstract (English)
Diffusion models have become a standard approach for generative modeling in continuous domains, yet their application to discrete data remains challenging. We investigate why Gaussian diffusion models with the DDPM solver struggle to sample from discrete distributions that are represented as a mixture of delta-distributions in the continuous space. Using a toy Random Hierarchy Model, we identify a critical sampling interval in which the density of noisified data becomes multimodal. In this regime, DDPM occasionally enters low-density regions between modes producing out-of-distribution inputs for the model and degrading sample quality. We show that existing heuristics, including self-conditioning and a solver we term q-sampling, help alleviate this issue. Furthermore, we demonstrate that combining self-conditioning with switching from DDPM to q-sampling within the critical interval improves generation quality on real data. We validate these findings across conditional and unconditional tasks in multiple domains, including text, programming code, and proteins.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。