通过限制注意力范围，让扩散语言模型推理快61倍且不丢精度。
DPad: Efficient Diffusion Language Models with Suffix Dropout
- 只关注附近有限的后续词元，用滑动窗口+距离衰减丢弃策略
- 在多个模型上实现最高61.4倍推理加速，准确率几乎不变
- 无需训练、兼容现有优化，几行代码就能接入
基于扩散的大型语言模型（dLLMs）将解码过程视为去噪过程，从而并行生成文本，但因每步需预测所有未来词元而计算开销高，仅保留少量有效信息。本文提出无需训练的Diffusion Scratchpad（DPad）方法，通过限制注意力范围至少量邻近词元，既保持生成质量又消除冗余。DPad结合两种策略：(i) 滑动窗口，维持固定长度的后续词元窗口；(ii) 距离衰减丢弃，在注意力计算前确定性移除远距离词元。该设计兼容前缀缓存等现有优化，仅需少量代码即可实现。在LLaDA-1.5与Dream模型上的多基准评估显示，DPad相较原始dLLMs最高可实现61.4倍加速，同时保持相当的生成准确性，展现出高效可扩展长序列推理的巨大潜力。代码已开源于https://github.com/Crys-Chen/DPad。
原文摘要 · Abstract (English)
Diffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction. We propose Diffusion Scratchpad (DPad), a training-free method that restricts attention to a small set of nearby suffix tokens, preserving fidelity while eliminating redundancy. DPad integrates two strategies: (i) a sliding window, which maintains a fixed-length suffix window, and (ii) distance-decay dropout, which deterministically removes distant suffix tokens before attention computation. This simple design is compatible with existing optimizations such as prefix caching and can be implemented with only a few lines of code. Comprehensive evaluations across multiple benchmarks on LLaDA-1.5 and Dream models demonstrate that DPad delivers up to $\mathbf{61.4\times}$ speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference. Our code is available at https://github.com/Crys-Chen/DPad.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。