通过复用注意力激活提升扩散语言模型推理效率
DARE: Diffusion Language Model Activation Reuse for Efficient Inference
- 发现扩散模型中令牌间注意力激活高度冗余，利用查询变化预测冗余
- 提出DARE-KV和DARE-O机制，最高复用87%注意力激活，降低1.20倍每层延迟
- 效果显著且兼容现有优化技术，适合关注推理加速的研究者
扩散大语言模型（dLLMs）作为自回归模型的潜在替代方案，具备更强表达能力与并行生成潜力，但开源版本在效率和质量上仍落后。我们发现dLLMs中存在未被充分挖掘的特性：双向自注意力中的令牌级冗余。自注意力激活在不同令牌间高度相关，查询表示的时间变化可预测对应键、值及输出激活的冗余。为此提出DARE，包含两个互补机制：DARE-KV复用缓存的键值激活，DARE-O复用输出激活，以减少冗余计算且保持生成质量。DARE实现最高达1.20倍每层延迟降低，最多复用87%注意力激活，推理与代码生成基准上性能下降极小，平均仅2.0%（DARE-KV）和1.2%（DARE-O）。结合前缀缓存与Fast-dLLM等技术可实现叠加增益，无需重新训练。结果表明令牌级复用是提升扩散类大模型效率的有效策略，同时保障生成保真度。代码已开源。
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to auto-regressive (AR) models, offering greater expressive capacity and potential for parallel generation and faster inference. However, open-source dLLMs remain immature, lagging behind AR models in both efficiency and quality. We identify an underexplored property of dLLMs: *token-wise redundancy* in bi-directional self-attention. Self-attention activations are highly correlated across tokens, and temporal changes in query representations can predict redundancy in corresponding key, value, and output activations. We introduce DARE, with two complementary mechanisms: DARE-KV, which reuses cached key-value (KV) activations, and DARE-O, which reuses output activations to reduce redundant computation while preserving quality. DARE achieves up to 1.20x per-layer latency reduction and reuses up to 87% of attention activations, with negligible degradation on reasoning and code-generation benchmarks. DARE-KV and DARE-O incur average performance drops of only 2.0% and 1.2%, respectively. Combined with techniques such as prefix caching and Fast-dLLM, DARE provides additive gains without retraining. These results establish token-wise reuse as an effective strategy for improving the efficiency of diffusion-based LLMs while preserving generation fidelity. Code: https://github.com/enyac-group/DARE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。