提出无需修改模型的并行解码方法，实现扩散语言模型零损失加速。
Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Models
- 通过候选生成与验证的并行算法，减少模型前向传播次数。
- 在多个基准上实现最高2.83倍加速，性能零下降。
- 适合追求高效推理的扩散语言模型研究者与应用开发者。
扩散语言模型（DLLMs）作为超越自回归文本预测的新范式，具备通过并行解码实现多标记预测的潜力，从而提升推理效率。然而，高质量生成通常需要与序列长度相同的解码步数，导致每次仅生成一个标记，而现有并行解码算法因生成路径次优，虽带来加速但造成显著性能下降。为此，本文提出Free Draft-and-Verification（FreeDave），一种专为DLLMs设计的新型快速解码算法，可在不修改模型或添加额外模块的情况下实现无损并行解码。该算法基于并行生成候选与验证机制，理论上使用最少的模型前向调用即可复现逐标记解码的结果。在多个数学推理和代码生成基准上对不同DLLMs进行广泛评估，结果表明FreeDave可实现最高2.83倍的推理加速，且性能无下降。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLLMs) have emerged as a new paradigm of language modeling beyond autoregressive next-token prediction. Taking advantage of their inherent modeling foundations, DLLMs have the great potential of efficient inference with parallel decoding algorithms, which enable multi-token prediction. However, the high generation quality often requires the number of decoding steps equal to the sequence length, which performs a one-token-per-step decoding, and existing parallel decoding algorithms, which yield suboptimal decoding paths, bring inference speedup at the cost of non-negligible performance degradation. To overcome this challenge, we introduce Free Draft-and-Verification (FreeDave), a novel fast decoding algorithm tailored for DLLMs that achieves lossless parallel decoding without any model modification or extra modules. Specifically, we propose an algorithm of parallel-decoded candidate generation and verification, which is theoretically guaranteed to use the fewest model forward calls to reproduce the same sequence generated by one-token-per-step decoding. By extensive evaluations on math reasoning and code generation benchmarks across different DLLMs, FreeDave is proven to accelerate the inference up to $2.83\times$ without performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。