arXiv:2607.24507cs.LGcs.AI2026-07

将GPT-2直接适配到统一噪声扩散模型,提升生成质量与推理效率。

UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

论文配图:UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective
图 1 · 摘自论文原文
  • 统一反向速率目标,实现AR模型到离散扩散的无缝转换。
  • 在256步采样下,小/大模型分别达到97.783/5.2626和71.516/5.6669的最优指标组合。
  • 支持掩码与均匀噪声双模式切换,适合需要高可控性的生成任务。

现有方法多将自回归语言模型适配至掩码扩散,而本文直接将其适配到均匀噪声扩散,使每个词元在采样过程中均可编辑。然而,跨不同扰动核的适配仍具挑战,因现有扩散语言模型采用不同目标函数与预测参数化方式。本文通过将SEDD、MDLM/GIDD、M2S与Neural CTMC的条件损失统一为单一广义KL目标,揭示其内在联系,并推导出从干净词元预测到具体得分、后验均值及退出率/跳跃参数化的转换关系,构建共享的 $x_0$ 接口,支持掩码与均匀核间的自由切换。基于此,我们提出 extsc{UNIFUSION},一种简单持续预训练方法,可直接将预训练GPT2模型适配至均匀噪声扩散。通过对124M与355M参数模型系统评估,结果表明,随着采样预算从16步增至256步, extsc{UNIFUSION} 持续优化生成困惑度(GenPPL)与一元熵的权衡。在256步时, extsc{UNIFUSION}-S与 extsc{UNIFUSION}-M分别达到97.783/5.2626与71.516/5.6669的最优组合;在同规模模型中,无其他模型能同时超越其在两项指标上的表现。此外,在WinoGrande、SIQA与BBH三项任务上, extsc{UNIFUSION} 在对比模型中均取得最高准确率。

原文摘要 · Abstract (English)

Existing methods mainly adapt pretrained autoregressive (AR) language models to masked diffusion, whereas we directly adapt them to uniform-noise diffusion, where every token remains editable during sampling. However, adapting AR checkpoints across corruption kernels remains challenging because existing DLMs use different objectives and prediction parameterizations. We establish connections among SEDD, MDLM/GIDD, M2S, and Neural CTMC by expressing their conditional losses as a single generalized Kullback--Leibler objective over model reverse rates. We further derive conversions from clean-token predictions to concrete-score, posterior-mean, and exit-rate/jump parameterizations, yielding a shared \(x_0\) interface that supports switching between mask and uniform kernels. Building on these connections, we propose \ours{}, a simple continual pre-training approach for directly adapting pretrained GPT2 checkpoints to uniform-noise diffusion. Through systematic evaluation of 124M- and 355M-parameter models, we show that \ours{} steadily improves the trade-off between generative perplexity (GenPPL) and unigram entropy as the sampling budget increases from 16 to 256 steps. At 256 steps, \ours{}-S and \ours{}-M achieve GenPPL/entropy pairs of \(97.783/5.2626\) and \(71.516/5.6669\), respectively; no evaluated model at the same scale simultaneously outperforms \ours{} on both metrics. At both scales, \ours{} also achieves the highest WinoGrande, SIQA, and BBH accuracy among the compared diffusion models.

扩散模型自回归语言生成统一建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。