通过信心引导聚焦，让长文本扩散模型推理快30倍且无损失。
Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing
- 利用相邻步骤的词元置信度相关性，预测未掩码区域以指导注意力稀疏化。
- 在32K上下文长度下实现超过29倍的速度提升，且无精度损失。
- 无需重新训练，适合追求高效长文本生成的开发者使用。
扩散型大语言模型（dLLMs）在非自回归解码范式下具备强大的长上下文处理能力，但双向全注意力带来的巨大计算开销限制了推理效率。尽管稀疏注意力有潜力，现有方法仍效果不佳，原因在于需预估尚未解码词元的注意力重要性，而扩散过程中未掩码位置未知。本文提出Focus-dLLM，一种专为高效长上下文dLLM推理设计的免训练注意力稀疏化框架。基于相邻解码步骤间词元置信度强相关性的发现，我们设计了基于历史置信度的指示器来预测未掩码区域；在此基础上，提出感知“注意力汇点”的剪枝策略，精确估计并移除冗余注意力计算，同时保留关键影响的注意力汇点。为进一步降低开销，该策略复用各层识别出的汇点位置，利用观察到的跨层一致性。实验表明，本方法在32K上下文长度下实现超过29倍的无损加速。代码已公开于：https://github.com/Longxmas/Focus-dLLM
原文摘要 · Abstract (English)
Diffusion Large Language Models (dLLMs) deliver strong long-context processing capability in a non-autoregressive decoding paradigm. However, the considerable computational cost of bidirectional full attention limits the inference efficiency. Although sparse attention is promising, existing methods remain ineffective. This stems from the need to estimate attention importance for tokens yet to be decoded, while the unmasked token positions are unknown during diffusion. In this paper, we present Focus-dLLM, a novel training-free attention sparsification framework tailored for accurate and efficient long-context dLLM inference. Based on the finding that token confidence strongly correlates across adjacent steps, we first design a past confidence-guided indicator to predict unmasked regions. Built upon this, we propose a sink-aware pruning strategy to accurately estimate and remove redundant attention computation, while preserving highly influential attention sinks. To further reduce overhead, this strategy reuses identified sink locations across layers, leveraging the observed cross-layer consistency. Experimental results show that our method offers more than $29\times$ lossless speedup under $32K$ context length. The code is publicly available at: https://github.com/Longxmas/Focus-dLLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。