让扩散语言模型推理快过自回归模型，突破速度瓶颈。
Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
- 用离散扩散强迫策略，实现块内自回归与块间并行解码。
- 在GSM8K上比LLaMA3快2.5倍，比基线扩散模型快50倍。
- 适合追求高速生成且不牺牲质量的开发者和研究者。
扩散语言模型（dLLMs）作为文本生成的新范式，有望在单次迭代中解码多个词元，但现有开源dLLMs尚未在推理速度上超越同规模自回归（AR）模型。本文提出一种简单有效的策略——离散扩散强迫（D2F），使dLLMs具备两项关键能力：(1) 块内自回归生成，支持键值缓存利用；(2) 预测后续词元无需完成前一块，实现跨块并行解码。由此将原生dLLMs改造为兼具AR与扩散特性的混合推理架构。D2F通过基于预训练dLLMs的非对称蒸馏实现。我们还设计了流水线并行解码算法，可在效率与效果间灵活权衡。实验表明，采用D2F的dLLMs在GSM8K任务上推理速度超过LLaMA3和Qwen2.5达2.5倍以上；相较于LLaDA和Dream等基线模型，加速比可达50倍以上，同时保持相近生成质量。代码已开源：https://github.com/zhijie-group/Discrete-Diffusion-Forcing。
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs for text generation, with the potential to decode multiple tokens in a single iteration. However, none of the existing open-source dLLMs have achieved superior inference speed over AR LLMs of similar size. This paper breaks this barrier based on a simple and effective strategy named discrete diffusion forcing (D2F). D2F equips dLLMs with two key capabilities: (1) block-wise autoregressive generation to enable KV cache utilization; (2) prediction of following tokens without requiring completion of prior blocks for inter-block parallel decoding. In this way, the vanilla dLLMs are refurbished into an AR-diffusion hybrid paradigm for efficient inference. D2F can be implemented with an asymmetric distillation process based on pre-trained dLLMs. We further propose a pipelined parallel decoding algorithm, which enables a trade-off between efficiency and efficacy. Empirically, D2F dLLMs achieve more than $\mathbf{2.5\times}$ inference speed than LLaMA3 and Qwen2.5 on GSM8K. Compared to vanilla dLLMs like LLaDA and Dream, the acceleration can be more than $\mathbf{50\times}$ while maintaining comparable output quality. The code is available at https://github.com/zhijie-group/Discrete-Diffusion-Forcing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。