提出无分解误差的离散扩散语言模型，提升生成准确率与速度。
Factorization-Error-Free Discrete Diffusion Language Model via Speculative Decoding
- 用精确前缀条件分解替代独立预测，保留词元依赖关系。
- 在扩散去噪中引入推测解码，平均提速3.86倍，准确率提升5.04个百分点。
- 适合追求高精度与高速推理的生成任务，如代码生成与数学推理。
离散扩散语言模型通过并行预测提高生成效率，但标准的X₀预测方法因用独立词元分布近似干净词元后验，引入分解误差。本文提出无分解误差的离散扩散语言建模（FeF-DLLM），将独立的干净词元预测替换为精确的前缀条件化后验分解，更好保持词元间依赖。为降低前缀条件带来的串行开销，FeF-DLLM在扩散去噪中进一步融入推测解码，在保持DLLM并行预测和重掩码特性的同时加速推理。理论上证明了FeF-DLLM从真实联合分布生成，并推导其期望加速比。在GSM8K、MATH、HumanEval和MBPP数据集上的实验表明，该方法平均准确率提升5.04个百分点，推理速度平均提升3.86倍。
原文摘要 · Abstract (English)
Discrete diffusion language models improve generation efficiency through parallel token prediction, but standard $X_0$ prediction methods introduce factorization errors by approximating the clean token posterior with independent token-wise distributions. This paper proposes Factorization-Error-Free Discrete Diffusion Language Modeling (FeF-DLLM), which replaces independent clean-token prediction with an exact prefix-conditioned factorization of the clean posterior to better preserve token dependencies. To reduce the sequential cost introduced by prefix conditioning, FeF-DLLM further incorporates speculative decoding within diffusion denoising, accelerating inference while maintaining the parallel prediction and re-masking properties of DLLMs. Theoretically, we prove that FeF-DLLM generates from the true joint distribution and derive its expected acceleration ratio. Experiments on GSM8K, MATH, HumanEval, and MBPP demonstrate that our method improves accuracy by an average of 5.04 percentage points while achieving an average inference speedup of $3.86\times$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。