通过一致性建模与块级因果掩码，让扩散语言模型生成更快更高效。
CDLM: Consistency Diffusion Language Models For Faster Sampling
- 引入一致性建模，支持多标记一次性生成，大幅减少采样步数。
- 在数学和编码任务上实现3.6至14.5倍加速，保持高准确率。
- 支持标准键值缓存，适合实际部署场景的快速生成需求。
扩散语言模型（DLMs）提供了一种有前景的并行生成范式，但因需大量精炼步骤且无法使用标准键值缓存而导致推理缓慢。我们提出CDLM（一致性扩散语言模型），一种基于训练的加速方法，同时解决这两个瓶颈。CDLM通过引入一致性建模，显著减少所需采样步骤，实现多标记一次性完成生成；此外，在微调中强制使用块级因果注意力掩码，使模型完全兼容键值缓存。实验表明，CDLM在数学和编码任务上实现了3.6倍至14.5倍的延迟降低，同时保持竞争力的准确性。完整训练与评估代码已开源：https://github.com/SqueezeAILab/CDLM。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) offer a promising parallel generation paradigm but suffer from slow inference due to numerous refinement steps and the inability to use standard KV caching. We introduce CDLM (Consistency Diffusion Language Models), a training-based acceleration method that simultaneously tackles both bottlenecks. CDLM integrates consistency modeling to drastically reduce the number of required sampling steps by enabling multi-token finalization. Furthermore, we enforce a block-wise causal attention mask during fine-tuning, making the model fully compatible with KV caching. Experiments show CDLM achieves 3.6x-14.5x lower latency while maintaining competitive accuracy on math and coding tasks. The full training and evaluation code is available at https://github.com/SqueezeAILab/CDLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。