arXiv:2605.00161cs.LG2026-05

提出新方法让文本扩散模型更快生成高质量文本,少步数也能更好。

Consistent Diffusion Language Models

论文配图:Consistent Diffusion Language Models
图 1 · 摘自论文原文
  • 用路径不变性训练去噪器,跨噪声层级保持一致
  • 少至5步采样就超越多阶段模型,全场景表现领先
  • 无需预训练教师模型,适合快速部署的生成任务

扩散语言模型(DLMs)虽可并行生成,但高质量样本仍需数百步。连续空间中一致性训练能加速扩散,但在离散空间因缺乏对应的样本空间微分方程而难以直接应用。本文提出,离散扩散的合理替代是精确后验桥——即任意噪声水平间闭式条件分布,适用于掩码和均匀扩散等广泛扰动。基于此,提出多路径离散一致性(MPDC)原则,训练去噪器在期望上对这些随机桥路保持路径不变,并实现为单阶段训练的一致扩散语言模型(CDLM)。CDLM目标统一了掩码扩散、连续一致性模型及渐进或离散蒸馏,可视为其极限或近似。实验表明,CDLM在条件与无条件文本生成上均达新基准,始终优于强基线离散扩散模型,尤其在少步数下优势显著,最大提升出现在5步以内。结果表明CDLM是下一代高速高保真离散生成建模的可靠基础。

原文摘要 · Abstract (English)

Diffusion language models (DLMs) are an attractive alternative to autoregressive models because they promise sublinear-time, parallel generation, yet practical gains remain elusive as high-quality samples still demand hundreds of refinement steps. In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion. For discrete diffusion, no analogous sample-space ODE exists, making direct adaptation ill-defined. We argue that the right discrete substitute is the exact posterior bridge, the closed-form conditional law linking any two noise levels, which is available for broad corruptions including masked and uniform diffusion. Building on this observation, we introduce Multi-Path Discrete Consistency (MPDC), a new principle that trains a denoiser to be path-invariant in expectation across these stochastic bridges, and instantiate it as the Consistent Diffusion Language Model (CDLM), a single-stage training framework that does not require an already trained teacher model. Our CDLM objective recovers masked diffusion, continuous consistency models, and progressive or discrete distillation as analytic limits or empirical approximations of one common view. Empirically, CDLM establishes a new state of the art on both conditional and unconditional text-generation, consistently outperforming strong base discrete diffusion models and often even multi-stage distilled baselines across sampling budgets, with the largest gains in the few-step regime. Together, these results position CDLM as a principled and scalable foundation for the next generation of fast, high-fidelity discrete generative modeling.

扩散模型文本生成高效采样一致性训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。