利用推测解码的注意力结果，实现高效稀疏注意力，显著减少计算开销。
SpecAttn: Speculating Sparse Attention
- 复用推测解码中已计算的注意力权重，识别关键词元。
- 在PG-19上缓存访问减少75%以上，困惑度仅上升15.29%。
- 无需训练，可无缝集成到现有推理流程，适合大模型部署者。
大型语言模型在推理时因自注意力机制的二次复杂度面临严重计算瓶颈，尤其在上下文长度增加时更为显著。我们提出SpecAttn，一种无需训练的新方法，可无缝集成现有推测解码技术，实现预训练Transformer中的高效稀疏注意力。核心思路是利用推测解码中草稿模型已计算的注意力权重，识别对目标模型重要的词元，从而消除冗余计算并保持输出质量。SpecAttn采用三项关键技术：基于KL散度的草稿与目标模型层间对齐、基于GPU优化的无排序top-p词元选择算法，以及依据预测动态修剪键值缓存。通过复用标准推测解码流程中已有的计算工作，SpecAttn在PG-19数据集上实现超过75%的键值缓存访问减少，困惑度仅增加15.29%，显著优于现有稀疏注意力方法。该方法表明，推测执行可扩展为近似验证机制，且性能损失极小。
原文摘要 · Abstract (English)
Large Language Models (LLMs) face significant computational bottlenecks during inference due to the quadratic complexity of self-attention mechanisms, particularly as context lengths increase. We introduce SpecAttn, a novel training-free approach that seamlessly integrates with existing speculative decoding techniques to enable efficient sparse attention in pre-trained transformers. Our key insight is to exploit the attention weights already computed by the draft model during speculative decoding to identify important tokens for the target model, eliminating redundant computation while maintaining output quality. SpecAttn employs three core techniques: KL divergence-based layer alignment between draft and target models, a GPU-optimized sorting-free algorithm for top-p token selection from draft attention patterns, and dynamic key-value cache pruning guided by these predictions. By leveraging the computational work already performed in standard speculative decoding pipelines, SpecAttn achieves over 75% reduction in key-value cache accesses with a mere 15.29% increase in perplexity on the PG-19 dataset, significantly outperforming existing sparse attention methods. Our approach demonstrates that speculative execution can be enhanced to provide approximate verification without significant performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。