arXiv:2601.02076cs.CLcs.AI2026-01被引 11

提出延迟承诺解码,提升扩散语言模型生成质量与效率。

Deferred Commitment Decoding for Diffusion Language Models

  • 用滑动窗口按置信度分批解码,低不确定性词提前确定
  • 在多个任务上平均提升生成准确率1.73%,最高达16.5%
  • 无需训练,兼容缓存机制,适合需要精确推理的任务

扩散语言模型(DLMs)作为自回归模型的有力替代,实现了并行文本生成。为提升推理效率和KV缓存兼容性,现有方法普遍采用分块扩散,按块解码。然而,这种范式存在结构性缺陷,即边界诱导上下文截断(BICT):靠近块边界的未解码词被迫提前确定,即使附近未来上下文可显著降低不确定性。这导致解码置信度下降,生成质量变差,尤其在数学求解和代码生成等需精确推理的任务中表现更差。本文提出无训练的延迟承诺解码(DCD),通过维护一个基于置信度的滑动窗口,在掩码词上优先解析低不确定性词,推迟高不确定性词的决策,直到获得足够上下文证据。在多个扩散语言模型、基准测试及缓存配置下的实验表明,相比固定分块扩散方法,DCD平均提升生成准确率1.73%,最显著提升达16.5%,证明根据不确定性延迟提交决策是提升扩散语言模型解码质量和效率的简单而有效原则。

原文摘要 · Abstract (English)

Diffusion language models (DLMs) have recently emerged as a strong alternative to autoregressive models by enabling parallel text generation. To improve inference efficiency and KV-cache compatibility, prior work commonly adopts block-based diffusion, decoding tokens block by block. However, this paradigm suffers from a structural limitation that we term Boundary-Induced Context Truncation (BICT): undecoded tokens near block boundaries are forced to commit without access to nearby future context, even when such context could substantially reduce uncertainty. This limitation degrades decoding certainty and generation quality, especially for tasks requiring precise reasoning, such as mathematical problem solving and code generation. We propose Deferred Commitment Decoding (DCD), a novel, training-free decoding strategy that mitigates this issue. DCD maintains a certainty-aware sliding window over masked tokens, resolving low-uncertainty tokens early while deferring high-uncertainty tokens until sufficient contextual evidence becomes available. Extensive experiments across multiple diffusion language models, benchmarks, and caching configurations show that DCD improves generation accuracy by 1.73% with comparable time on average compared to fixed block-based diffusion methods, with the most significant improvement reaching 16.5%. These results demonstrate that deferring token commitment based on uncertainty is a simple yet effective principle for improving both the quality and efficiency of diffusion language model decoding.

扩散模型文本生成解码优化推理效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。