arXiv:2602.06953cs.CL2026-02被引 11

让扩散语言模型更快生成,且不降低质量。

DAWN: Dependency-Aware Fast Inference for Diffusion LLMs

  • 基于依赖图动态选择解码位置,提升并行效率。
  • 实测速度提升1.80-8.06倍,生成质量几乎不变。
  • 无需训练,适合追求高效推理的开发者使用。

扩散大语言模型(dLLMs)在文本生成中表现出色,尤其得益于其固有的并行解码能力。然而,受限于质量与速度的权衡,现有推理方案采用保守的并行策略,未能充分挖掘效率潜力。核心挑战在于:并行解码假设各位置可独立填充,但实际中词元常存在语义耦合,某一位置的正确选择会限制其他位置的有效选择。若不建模这种跨词元依赖,会导致输出质量下降。为此,我们提出DAWN——一种无需训练、依赖感知的快速推理方法。DAWN通过提取词元依赖关系,利用两个关键洞察:(1) 受未掩码位置影响的位置更具可靠性;(2) 同时解码强耦合且不确定的位置会引入错误。基于此,DAWN利用依赖图在每轮迭代中选择更可靠的解码位置,实现高并行度且生成质量损失极小。在多个模型和数据集上的实验表明,相较于基线,DAWN将推理速度提升1.80-8.06倍,同时保持生成质量。代码已开源:https://github.com/lizhuo-luo/DAWN。

原文摘要 · Abstract (English)

Diffusion large language models (dLLMs) have shown advantages in text generation, particularly due to their inherent ability for parallel decoding. However, constrained by the quality--speed trade-off, existing inference solutions adopt conservative parallel strategies, leaving substantial efficiency potential underexplored. A core challenge is that parallel decoding assumes each position can be filled independently, but tokens are often semantically coupled. Thus, the correct choice at one position constrains valid choices at others. Without modeling these inter-token dependencies, parallel strategies produce deteriorated outputs. Motivated by this insight, we propose DAWN, a training-free, dependency-aware decoding method for fast dLLM inference. DAWN extracts token dependencies and leverages two key motivations: (1) positions dependent on unmasked certain positions become more reliable, (2) simultaneously unmasking strongly coupled uncertain positions induces errors. Given those findings, DAWN leverages a dependency graph to select more reliable unmasking positions at each iteration, achieving high parallelism with negligible loss in generation quality. Extensive experiments across multiple models and datasets demonstrate that DAWN speedups the inference by 1.80-8.06x over baselines while preserving the generation quality. Code is released at https://github.com/lizhuo-luo/DAWN.

扩散模型推理加速依赖建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。