arXiv:2410.01949cs.LG2024-10ICLR被引 47

用耦合模型补足离散扩散模型的变量依赖缺陷,大幅减少采样步数。

Discrete Copula Diffusion

  • 引入耦合模型补充扩散过程中的变量依赖信息
  • 在8到32倍更少步骤下实现更优文本生成效果
  • 无需微调即可融合,适合追求高效生成的研究者

离散扩散模型在建模自然语言和DNA序列等复杂数据方面取得显著进展,但与连续数据的扩散模型不同,其仍需数百甚至上千步才能生成高质量样本。本文揭示其性能受限的根本原因——在每一步去噪中无法捕捉输出变量间的依赖关系。为此,提出一种通用方法,通过引入称为耦合模型的深层生成模型来补充缺失的依赖信息。该方法无需微调扩散模型或耦合模型,即可显著减少去噪步数并实现高质量生成。将此方法应用于自回归耦合模型时,混合模型在无条件与条件文本生成上均优于任一单独模型,仅需原扩散模型8至32倍更少的去噪步骤即达到更好效果。本研究不仅提出有效的离散扩散生成算法,更强调了建模变量间依赖对提升性能的关键作用。

原文摘要 · Abstract (English)

Discrete diffusion models have recently shown significant progress in modeling complex data, such as natural languages and DNA sequences. However, unlike diffusion models for continuous data, which can generate high-quality samples in just a few denoising steps, modern discrete diffusion models still require hundreds or even thousands of denoising steps to perform well. In this paper, we identify a fundamental limitation that prevents discrete diffusion models from achieving strong performance with fewer steps -- they fail to capture dependencies between output variables at each denoising step. To address this issue, we provide a formal explanation and introduce a general approach to supplement the missing dependency information by incorporating another deep generative model, termed the copula model. Our method does not require fine-tuning either the diffusion model or the copula model, yet it enables high-quality sample generation with significantly fewer denoising steps. When we apply this approach to autoregressive copula models, the combined model outperforms both models individually in unconditional and conditional text generation. Specifically, the hybrid model achieves better (un)conditional text generation using 8 to 32 times fewer denoising steps than the diffusion model alone. In addition to presenting an effective discrete diffusion generation algorithm, this paper emphasizes the importance of modeling inter-variable dependencies in discrete diffusion.

离散扩散耦合模型文本生成高效采样

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。