提出双模型推测解码，让扩散模型生成更快更准。
DualDiffusion: A Speculative Decoding Strategy for Masked Diffusion Models
- 用轻量草稿模型快速生成，再用精确验证模型纠错。
- 在MMLU和GSM8K上减少生成步数，保持高准确率。
- 适合追求高效高质量文本生成的研究者与开发者。
掩码扩散模型（MDMs）通过并行生成和双向上下文建模，为自回归语言模型提供了有前景的替代方案。然而，由于双向注意力无法缓存键值对，推理速度受限于每步需$O(N^2)$计算。现有方法如FastDLLM和DkvCache虽通过注意力近似和缓存策略提升速度，但以牺牲生成质量为代价。本文提出DualDiffusion，一种面向MDMs的推测解码框架，结合快速草稿模型（使用高效近似）与更慢但更准确的验证模型。通过运行多步轻量草稿后仅一步验证，该方法在生成步数与准确性之间实现了更优权衡。我们在MMLU和GSM8K数据集上验证了该方法，结果表明DualDiffusion在保持高准确率的同时显著减少了所需生成步数，有效推动了掩码扩散语言模型的质量-效率曲线边界。
原文摘要 · Abstract (English)
Masked Diffusion Models (MDMs) offer a promising alternative to autoregressive language models by enabling parallel token generation and bidirectional context modeling. However, their inference speed is significantly limited by the inability to cache key-value pairs due to bidirectional attention, requiring $O(N^2)$ computations at each generation step. While recent methods like FastDLLM and DkvCache improve inference speed through attention approximations and caching strategies, they achieve speedups at the cost of generation quality. We propose DualDiffusion, a speculative decoding framework for MDMs that combines fast drafter models (using efficient approximations) with slower, more accurate verifier models. By running multiple steps of a lightweight drafter followed by a single verification step, DualDiffusion achieves a superior Pareto frontier between generation steps and accuracy compared to existing approaches. We evaluate our method on MMLU and GSM8K, demonstrating that DualDiffusion maintains high accuracy while reducing the number of generation steps required, effectively pushing the quality-efficiency trade-off curve for masked diffusion language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。