让扩散语言模型自己教自己高效生成，提升速度与质量。
Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers
- 提出可撤销的并行解码方法，动态修正不靠谱的生成结果。
- 在GSM8K上准确率提至76.58%，推理步数减少6.83倍。
- 适合追求高效高质生成的AI研发者与系统优化人员。
扩散大语言模型（DLLMs）虽具并行生成优势，但开源模型仍面临质量与速度的权衡：加速解码时多步暴露令牌常导致严重质量下降。我们归因于训练与推理间的不匹配，且被不可逆解码加剧。训练中从随机破坏状态重建令牌，而高效推理需自适应去噪顺序——先处理简单令牌，延后依赖上下文的。为此，我们提出两种互补方法：推理端的可撤销并行解码，以及训练端注入该过程揭示的可靠顺序。首先提出无需训练的WINO算法，主动生成多个令牌，用全局上下文验证并重掩不可靠项。基于此发现的顺序，进一步引入WINO+，将验证后的去噪轨迹注入模型参数，使训练对齐高效推理。实验表明，WINO在LLaDA和MMaDA上同时提升质量和效率；在GSM8K上，准确率从73.24%升至75.82%，步数减少6.10倍；WINO+进一步达76.58%，步数减少6.83倍；在Flickr30K上，步数减少16.22倍，CIDEr得分提升。结果证明，DLLMs可通过可撤销解码发现可靠去噪顺序，并学习遵循以实现更快生成。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLLMs) promise fast parallel generation, yet open-source DLLMs still face a severe quality-speed trade-off: accelerating decoding by revealing multiple tokens often causes substantial quality degradation. We attribute this dilemma to a train-inference mismatch amplified by irreversible decoding. While training reconstructs tokens from randomly corrupted states, efficient inference requires an adaptive denoising order, where easier tokens are revealed earlier and context-dependent ones are deferred. This view motivates two complementary methods: an inference-time method that makes parallel decoding revokable, and a training-time extension that distills the reliable order exposed by this revokable process. Accordingly, we first propose Wide-In, Narrow-Out (WINO), a training-free decoding algorithm that enables revokable parallel generation. WINO aggressively drafts multiple tokens, verifies generated tokens with enriched global context, and re-masks unreliable ones for later refinement. Building on this discovered order, we further introduce WINO+, which injects the verified denoising trajectories produced by WINO into model parameters, aligning training with efficient inference. Experiments on LLaDA and MMaDA show that WINO improves both quality and efficiency, while WINO+ further strengthens this progression. On GSM8K, WINO improves accuracy from 73.24% to 75.82% with a 6.10x step reduction, and WINO+ further achieves 76.58% with a 6.83x reduction. On Flickr30K, WINO+ reaches a 16.22x step reduction with improved CIDEr. These results demonstrate that DLLMs can serve as their own efficiency teachers by first discovering reliable denoising orders through revokable decoding and then learning to follow them for faster generation. Code is available at https://github.com/Feng-Hong/WINO-DLLM/tree/WINO-plus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。