arXiv:2604.17068cs.CLcs.LG2026-04

通过动态评估生成稳定性,提升扩散语言模型的生成准确性

Stability-Weighted Decoding for Diffusion Language Models

论文配图:Stability-Weighted Decoding for Diffusion Language Models
图 1 · 摘自论文原文
  • 引入时序稳定性度量,基于分布变化判断是否该解码
  • 在代码与数学推理任务中显著提升准确率,加速比下仍保持优势
  • 无需训练、即插即用,适配多种解码策略

扩散大语言模型(dLLMs)通过迭代去噪全掩码序列实现并行文本生成,每步解码部分掩码词元。现有解码策略依赖单步静态置信度,忽略时间演化,常导致不稳定的词元过早解码。本文理论证明:词元的时序不稳定性(连续预测分布间的KL散度)是其与剩余掩码上下文互信息的严格下界,表明不稳定词元不宜提前解码。基于此,提出无训练、可插拔的稳定性加权解码(SWD),将时序稳定性融入词元评分,作为通用调节器适配任意基于分数的解码策略。在代码生成与数学推理基准上的实验表明,SWD在多种评分指标与选择策略下均一致提升生成准确率,且在不同加速比下表现鲁棒,显著优于标准基线。

原文摘要 · Abstract (English)

Diffusion large language models (dLLMs) enable parallel text generation by iteratively denoising a fully masked sequence, unmasking a subset of masked tokens at each step. Existing decoding strategies rely on static confidence metrics computed at a single denoising step, ignoring temporal history and often leading to premature unmasking of unstable tokens. In this work, we theoretically establish that a token's temporal instability, quantified by the KL divergence between consecutive prediction distributions, provides a strict lower bound on its mutual information with the remaining masked context, indicating that temporally unstable tokens are inherently unsafe to unmask. Based on this insight, we propose Stability-Weighted Decoding (SWD), a training-free, plug-and-play strategy that incorporates temporal stability into token scoring and acts as a universal modulator for arbitrary score-based decoding policies. Experiments on code generation and mathematical reasoning benchmarks demonstrate that SWD consistently improves generation accuracy across representative scoring metrics and selection policies, and exhibits exceptional robustness, maintaining a significant performance lead over standard baselines across varying acceleration ratios.

扩散模型语言模型解码优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。