arXiv:2606.09159cs.CLcs.AI2026-06

提出统一能量函数,提升扩散语言模型并行生成效果

Unified Energy for Invariant and Independent Decoding in Diffusion Language Models

  • 设计统一能量函数,同时解决不变性与独立性问题
  • 在多个模型上实现接近自回归基线的生成质量
  • 无需采样估算,可适配任意规模模型,适合研究并行生成

扩散语言模型(DLMs)通过迭代去噪全序列实现并行文本生成,相比自回归解码更具灵活性。然而现有方法未能充分捕捉词元间关系,导致并行度越高性能越落后于自回归基线。本文系统分析该差距,识别出三个关键因素:(i) 模型容量,(ii) 依赖关系,(iii) 不变性。为此,我们首先提出不变能量(Inv-E)及其基于采样的有效估计器以处理不变性问题;进一步结合独立能量(Ind-E),构建统一能量(Uni-E),全面涵盖上述因素。Uni-E 的独特优势在于可精确计算,无需采样估计分区。此外,Uni-E 具有模型无关性,可扩展至任意规模模型。我们还证明了 Uni-E 可纠正由依赖性和不变性引起的分布偏移。在 DLMs 与 DLLMs 上的大量实验验证了 Uni-E 的有效性。

原文摘要 · Abstract (English)

Diffusion Language Models (DLMs) enable parallel text generation by iteratively denoising a full sequence, offering attractive flexibility compared to auto-regressive (AR) decoding. However, existing methods fail to fully capture token relationships, leading to a performance gap relative to AR baselines, especially as the degree of parallelism increases. In this paper, we give a systematic analysis of the gap, identifying three key factors: (i) model capacity, (ii) dependency, and (iii) invariance. To address these issues, we first propose an invariant energy (Inv-E) together with an effective sampling-based estimator to handle the invariance issue. By further combining with the independent energy (Ind-E), we obtain a unified energy (Uni-E), that accounts for all these factors. Uni-E enjoys a unique advantage: it can be computed exactly without sampling-based partition estimation. Besides, Uni-E is model agnostic and can therefore be scaled to models of arbitrary size. We further prove that Uni-E can correct the distribution shift caused by dependency and invariance. Extensive experiments across Diffusion Language Models (DLMs) and Diffusion Large Language Models (DLLMs) demonstrate the effectiveness of the proposed Uni-E.

扩散模型语言建模并行生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。