根据注意力熵动态分配计算，提升长文本大模型推理速度。
From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs
- 按注意力头和输入段的熵变化动态分配计算资源
- 在超10万词长文本上实现2.39倍加速，质量损失极小
- 无需训练，适合需要高效推理的长文本场景
现有稀疏注意力与键值缓存压缩方法通常对所有注意力头采用固定稀疏模式或统一预算，忽视了头与上下文间注意力行为的显著差异。我们观察到两类注意力头熵模式：熵始终接近零的刚性头，以及熵剧烈波动的动态头。关键在于，这类分布依赖上下文且无法离线预设。为此，我们提出EntropyInfer，一种无需训练的框架，通过注意力熵在prefilling阶段自适应分配每个头和段的计算量。在解码阶段，提出基于生成输出令牌的潜在键值缓存压缩方案，而非仅依赖prefill令牌，以识别并保留最关键的缓存条目。在Llama、Qwen及openPangu系列模型上的大量实验表明，EntropyInfer持续优于SnapKV、AdaKV与CritiPrefill等基线，在超过10万词长度下实现最高达2.39倍的端到端加速，同时相比全注意力的质量损失极小。代码已开源：https://github.com/SHA-4096/EntropyInfer。
原文摘要 · Abstract (English)
Existing sparse attention and KV cache compression methods for long-context LLM inference typically apply fixed sparsity patterns or uniform budgets across all attention heads, overlooking the substantial variation in attention behavior among heads and contexts. We observe two distinct entropy patterns among attention heads: Rigid Heads, whose entropy stays near zero across input segments, and Dynamic Heads, whose entropy fluctuates significantly. Crucially, the distribution of these types is context-dependent and cannot be predetermined offline. We therefore propose EntropyInfer, a training-free framework that uses attention entropy to adaptively allocate compute at the granularity of individual heads and segments during prefilling. For decoding, we introduce a latent KV cache compression scheme that leverages generated output tokens, rather than prefill tokens alone, to identify and retain the most critical cache entries. Extensive experiments on Llama, Qwen and openPangu model series show that EntropyInfer consistently outperforms baselines including SnapKV, AdaKV, and CritiPrefill, achieving up to 2.39$\times$ end-to-end speedup beyond 100k tokens with minimal quality degradation compared to full attention. The code is released in https://github.com/SHA-4096/EntropyInfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。