arXiv:2606.26120cs.CLcs.LG2026-06被引 5

动态调整缓存与并行解码,让扩散语言模型跑得更快不丢质量。

Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM

论文配图:Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM
图 1 · 摘自论文原文
  • 根据层间令牌动态分配缓存更新预算,智能优化资源使用。
  • 解码阈值自适应调节,在速度与生成质量间实现3倍以上提速。
  • 无需训练的即插即用方案,适合长文本生成场景部署。

扩散大语言模型(dLLMs)凭借双向注意力机制在文本生成中表现优异,但其计算复杂度随序列长度L呈L³增长,难以满足长序列与实时应用需求,主要因缺乏键值缓存兼容性及去噪步骤的非自回归特性。现有加速方法依赖静态缓存或并行解码策略,未能考虑不同层与解码步中令牌属性的动态变化。本文提出Dynamic-dLLM,一种无训练框架,包含两项核心技术:动态缓存更新(DCU),根据层间令牌动态自适应分配缓存更新预算;自适应并行解码(APD),动态校准解码阈值以平衡生成质量与效率。在LLaDA-8B-Instruct、LLaDA-1.5和Dream-v0-7B-Instruct等模型上,于MMLU、GSM8K和HumanEval等基准测试中验证,Dynamic-dLLM平均加速超过3倍,性能保持不变,显著优于现有加速方法,为dLLM高效部署提供即插即用方案。代码已开源。

原文摘要 · Abstract (English)

Diffusion Large Language Models (dLLMs) offer a promising alternative to autoregressive models, excelling in text generation tasks due to their bidirectional attention mechanisms. However, their computational complexity scales on the order of L cubed with the sequence length L. This poses significant challenges for long-sequence and real-time applications, primarily due to the lack of compatibility with key-value caching and the non-autoregressive nature of denoising steps. Existing acceleration methods rely on static caching or parallel decoding strategies, which fail to account for the dynamic behavior of token properties across layers and decoding steps. We propose Dynamic-dLLM, a training-free framework that enhances dLLM inference efficiency through two components: Dynamic Cache Updating (DCU), which adaptively allocates cache-update budgets based on layer-wise token dynamics, and Adaptive Parallel Decoding (APD), which dynamically calibrates decoding thresholds to balance generation quality and efficiency. Extensive experiments on models like LLaDA-8B-Instruct, LLaDA-1.5, and Dream-v0-7B-Instruct across benchmarks such as MMLU, GSM8K, and HumanEval demonstrate that Dynamic-dLLM significantly improves inference speed. It attains an average speedup exceeding 3 times while maintaining performance. Dynamic-dLLM outperforms state-of-the-art acceleration methods and provides a plug-and-play solution for efficient dLLM deployment without compromising performance. The code is available at https://github.com/TianyiWu233/DYNAMIC-DLLM.

扩散模型推理加速动态调度语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。