通过动态剔除冗余缓存，让扩散语言模型推理快10倍
Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction
- 基于注意力模式稳定性，动态保留关键词元，剔除无用缓存
- 在长文本任务中实现10倍吞吐提升，内存开销与原模型相当
- 无需训练，适合需要高速推理的长文本生成场景
扩散大语言模型（dLLMs）在推理和并行解码方面取得突破，但推理时存在二次方复杂度和高内存开销问题。现有缓存技术通过存储全层状态加速解码，却带来巨大内存占用，限制了长上下文应用。我们对dLLMs的注意力模式分析发现，跨层稀疏性显著：关键词元在多步解码中保持重要性，低相关性词元始终无关紧要，据此提出Sparse-dLLM——首个无需训练的动态缓存剔除框架，结合延迟双向稀疏缓存机制。利用词元重要性在步骤间的稳定性，该方法保留核心词元，并采用注意力引导策略动态剔除前后缀无用条目。在LLaDA和Dream系列模型上的大量实验表明，Sparse-dLLM相比原始dLLMs最高实现10倍吞吐提升，性能相近且峰值内存相似，效率与效果均优于现有方法。代码已开源。
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding but suffer from prohibitive quadratic computational complexity and memory overhead during inference. Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage that limit long-context applications. Our analysis of attention patterns in dLLMs reveals persistent cross-layer sparsity, with pivotal tokens remaining salient across decoding steps and low-relevance tokens staying unimportant, motivating selective cache eviction. We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention via delayed bidirectional sparse caching. By leveraging the stability of token saliency over steps, it retains critical tokens and dynamically evicts unimportant prefix/suffix entries using an attention-guided strategy. Extensive experiments on LLaDA and Dream series demonstrate Sparse-dLLM achieves up to 10$\times$ higher throughput than vanilla dLLMs, with comparable performance and similar peak memory costs, outperforming previous methods in efficiency and effectiveness. The code is available at https://github.com/OpenMOSS/Sparse-dLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。