arXiv:2502.00234cs.LGcs.CV2025-02NeurIPS被引 50

提出高阶算法加速离散扩散模型推理,提升生成质量与效率。

Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms

  • 引入高阶数值方法,实现更大步长下的精准推理。
  • 在数学推理、文本生成和图像合成任务中,相同计算量下样本质量显著优于现有方法。
  • 适用于从200M到8B参数模型,适合追求高效高质量生成的场景。

离散扩散模型在文本生成到图像合成等任务中表现优异,但其高维状态空间导致推理效率低下。现有方法分为精确采样与近似方法(如τ-跳变),前者耗时不可控且冗余计算多,后者仅具一阶精度。本文首次将高阶数值推理解法拓展至离散扩散模型,提出θ-梯形法,严格证明其在KL散度上具二阶精度。在GSM8K级数学推理、GPT-2级文本及ImageNet级图像生成任务上,该方法在相同计算成本下显著提升样本质量,且在200M至8B参数模型中均保持稳定优势。代码已开源:https://github.com/yuchen-zhu-zyc/DiscreteFastSolver。

原文摘要 · Abstract (English)

Discrete diffusion models have emerged as a powerful generative modeling framework for discrete data with successful applications spanning from text generation to image synthesis. However, their deployment faces challenges due to the high dimensionality of the state space, necessitating the development of efficient inference algorithms. Current inference approaches mainly fall into two categories: exact simulation and approximate methods such as $τ$-leaping. While exact methods suffer from unpredictable inference time and redundant function evaluations, $τ$-leaping is limited by its first-order accuracy. In this work, we advance the latter category by tailoring the first extension of high-order numerical inference schemes to discrete diffusion models, enabling larger step sizes while reducing error. We rigorously analyze the proposed schemes and establish the second-order accuracy of the $θ$-Trapezoidal method in KL divergence. Empirical evaluations on GSM8K-level math-reasoning, GPT-2-level text, and ImageNet-level image generation tasks demonstrate that our method achieves superior sample quality compared to existing approaches under equivalent computational constraints, with consistent performance gains across models ranging from 200M to 8B. Our code is available at https://github.com/yuchen-zhu-zyc/DiscreteFastSolver.

扩散模型高效推理高阶算法生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。