arXiv:2606.04236cs.CLcs.AI2026-06

通过动态选择支持性锚点,提升扩散语言模型并行解码效率与质量。

Supportive Token Revealing for Fast Diffusion Language Model Decoding

论文配图:Supportive Token Revealing for Fast Diffusion Language Model Decoding
图 1 · 摘自论文原文
  • 引入AXON模块,基于注意力与置信度信号选择关键支持性令牌
  • 在多个模型上减少函数评估次数,同时保持或提升生成准确率
  • 无需训练,适配现有并行解码策略,适合追求高效推理的开发者

离散扩散语言模型可通过并行更新多个掩码位置实现高效文本生成,但并行性带来质量与延迟的权衡。激进解码可能过早提交相互依赖的词元,而保守解码需更多去噪步骤。现有方法通过置信度或依赖性判断哪些词元安全释放,但避免不安全提交并不保证剩余序列易解码,因不确定词元可能依赖其他掩码词元,形成去噪瓶颈。我们提出AXON,一个无需训练的模块,可叠加于现有并行解码策略之上。不替代基础解码器,而是监控剩余不确定的掩码词元,在其状态表明需要额外上下文时介入。此时,决策标准从‘最安全释放的词元’转为‘最能支持后续去噪的确定性释放’。AXON利用注意力、不确定性与置信度信号选择锚点(即不确定位置所关注的确定性掩码词元)。在多个推理与代码生成基准上对多种扩散语言模型的实验表明,AXON显著改善了现有并行解码器的质量-延迟权衡,通常减少函数评估次数,同时维持或提升准确性。

原文摘要 · Abstract (English)

Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.

扩散模型并行解码语言模型推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。