让扩散语言模型实现高速并行推理,且保持生成质量。
SimSD: Simple Speculative Decoding in Diffusion Language Models

- 用插件式掩码策略让扩散模型具备可验证的逐词推测能力。
- 在四个基准上实现最高7.46倍吞吐提升,生成质量不降反升。
- 无需训练,可与缓存、分块解码等技术灵活组合使用。
扩散大语言模型(dLLMs)作为自回归(AR)模型的替代方案,通过并行或分块解码实现了更快的推理速度。然而,其基于掩码的语言建模形式与标准的逐词推测解码不兼容。在自回归解码中,因果掩码保留了时间有效的上下文,使目标模型能在单次前向传播中验证多个草稿词。而扩散模型依赖掩码标记和双向注意力,导致有效上下文随去噪步骤变化,无法直接进行逐词推测验证。为此,我们提出一种简单但高效的扩散语言模型推测解码算法SimSD,主要采用即插即用的掩码策略,为dLLMs引入具有时间一致性的逐词上下文。该方法显式引入草稿模型预测的参考标记,并设计注意力掩码以调控其与当前步标记的交互,使dLLMs能在单次前向传播中计算出草稿词的有效概率。这恢复了自回归模型中因果掩码提供的关键验证能力,同时保留了扩散模型的并行解码优势。所提方法无需训练,可灵活集成至其他加速技术如KV缓存和分块解码。在SDAR系列dLLMs上的实验表明,该方法在四个基准上实现了最高7.46倍的解码吞吐提升,且平均生成质量维持甚至提高。
原文摘要 · Abstract (English)
Diffusion large language models (dLLMs) have recently emerged as a promising alternative to autoregressive (AR) LLMs, offering faster inference through parallel or blockwise decoding. However, their masked language modeling formulation remains incompatible with standard token-level speculative decoding, one of the most effective acceleration techniques for AR models. In AR decoding, the causal mask preserves temporally valid token-level contexts, enabling a target model to verify multiple drafted tokens in a single forward pass. In contrast, dLLMs rely on mask tokens and bidirectional attention, causing the effective context to change across denoising steps and preventing direct token-level speculative verification. To bridge this gap, we propose a simple but effective speculative decoding algorithm for diffusion language models, named SimSD, which mainly adopts a plug-and-play masking strategy that equips dLLMs with temporally valid token-level contexts for speculative decoding. Our method explicitly introduces reference tokens from draft-model predictions and designs an attention mask that regulates their interaction with current-step tokens, allowing dLLMs to compute valid logits for drafted tokens in a single forward pass. This restores the key verification ability provided by causal masking in AR models while preserving the parallel decoding advantages of dLLMs. The proposed method is training-free and can be flexibly integrated with other acceleration techniques such as KV cache and blockwise decoding. Experiments on SDAR-family dLLMs across four benchmarks show that our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。