提出可撤销解码方法,让扩散语言模型更快更准。
Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs
- 并行草稿+双向验证,动态修正错误预测
- 数学题推理快6倍,准确率提升2.58%;图文生成提速10倍
- 无需重训练,适配现有扩散模型,适合追求效率的部署场景
扩散大型语言模型(DLLMs)作为自回归模型的高效替代方案,支持并行生成。然而现有方法存在严重的质量-速度权衡:越快的并行解码导致性能显著下降。我们发现根源在于标准解码不可逆,易受早期错误上下文影响而偏离正确方向。为此,提出无需训练的可撤销解码算法WINO,采用并行草稿与验证机制,在生成多个候选词的同时,利用模型双向上下文识别并重掩码可疑内容进行修正。在LLaDA和MMaDA等开源DLLMs上验证,效果显著:在GSM8K数学基准上,推理加速6倍且准确率提升2.58%;在Flickr30K图像描述任务中实现10倍加速且性能更高。更多实验进一步验证其优势并深入解析机制。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLLMs) have emerged as a compelling alternative to Autoregressive models, designed for fast parallel generation. However, existing DLLMs are plagued by a severe quality-speed trade-off, where faster parallel decoding leads to significant performance degradation. We attribute this to the irreversibility of standard decoding in DLLMs, which is easily polarized into the wrong decoding direction along with early error context accumulation. To resolve this, we introduce Wide-In, Narrow-Out (WINO), a training-free decoding algorithm that enables revokable decoding in DLLMs. WINO employs a parallel draft-and-verify mechanism, aggressively drafting multiple tokens while simultaneously using the model's bidirectional context to verify and re-mask suspicious ones for refinement. Verified in open-source DLLMs like LLaDA and MMaDA, WINO is shown to decisively improve the quality-speed trade-off. For instance, on the GSM8K math benchmark, it accelerates inference by 6$\times$ while improving accuracy by 2.58%; on Flickr30K captioning, it achieves a 10$\times$ speedup with higher performance. More comprehensive experiments are conducted to demonstrate the superiority and provide an in-depth understanding of WINO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。