提出DepCap,让扩散语言模型推理更快更准。
DepCap: Adaptive Block-Wise Parallel Decoding for Efficient Diffusion LM Inference

- 用前一块影响动态决定下一块边界,实现自适应分块。
- 识别无冲突词元并行解码,最高提速5.63倍且质量几乎不变。
- 无需训练,可直接接入现有模型,适合高效推理场景。
扩散语言模型(DLMs)因其并行解码和全局序列优化潜力,成为自回归生成的有力替代。为释放这一潜力,必须平衡生成质量与解码速度。现有分块式DLM解码方法通过分块顺序扩散提升效率,但多依赖固定分块策略或当前步局部信号确定分块边界,并采用保守的置信度判别进行并行解码,限制了性能提升。本文提出DepCap,一种无需训练的高效分块推理框架。其核心思想是:使用前一块的累积影响作为跨步信号以自适应确定下一块扩展范围;同时在每块内识别无冲突词元子集,实现安全并行解码。信息论分析表明,候选块受前一块影响近似可加性,支持该分块准则。实验显示,DepCap在多个DLM主干及推理与编程基准上均实现优异的速度-质量权衡,最高提速达5.63倍,且性能下降可忽略。
原文摘要 · Abstract (English)
Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive language generation due to their potential for parallel decoding and global refinement of the entire sequence. To unlock this potential, DLM inference must carefully balance generation quality and decoding speed. Recent block-wise DLM decoding methods improve this trade-off by performing diffusion-based decoding sequentially in blocks. However, existing methods typically rely on fixed block schedules or current-step local signals to determine block boundaries, and use conservative confidence-based parallel decoding to avoid conflicts, limiting the quality-speed trade-off. In this paper, we argue that block-wise DLM inference requires more suitable signals for its two core decisions: cross-step signals for determining block boundaries, and token-level conflict signals for parallel decoding. Based on this view, we propose DepCap, a training-free framework for efficient block-wise DLM inference. Specifically, DepCap instantiates the cross-step signal as the influence of the last decoded block and uses it to adaptively determine how far the next block should extend, while identifying a conflict-free subset of tokens for safe parallel decoding within each block, enabling substantial inference acceleration with negligible quality degradation. DepCap is a plug-and-play method applicable to various DLMs, and compatible with existing KV-cache strategies for block-wise DLM. An information-theoretic analysis further suggests that the cumulative last-block influence on a candidate block is approximately additive across tokens, supporting the proposed block-partitioning criterion. Experimental results show that DepCap achieves favorable speed-quality trade-offs across multiple DLM backbones and reasoning and coding benchmarks, with up to 5.63$\times$ speedup without significant performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。