优化长推理大模型的键值缓存，让计算效率提升17倍
Precomputing the Future-Offset Average in TriAttention
- 通过代数恒等式将17种未来距离的平均计算合并为一次预计算
- 评分成本从17次降至1次，但保留相同剪枝效果
- 适合关注缓存优化和计算效率的模型部署研究者
TriAttention是一种用于压缩长推理大模型键值缓存的方法：它根据每个缓存键未来可能接收注意力的程度进行评分，并淘汰得分最低的键。由于键无法预知未来查询的位置，评分需对17种可能的未来距离求平均。我们指出，该平均是免费的：未来距离仅通过位置相关的旋转影响评分，因此整个17倍平均可精确地简化为一个单次离线计算的每带权重。评分时只需一次评估，无需17次，且剪枝结果不变。该优化仅作用于评分计算阶段，不涉及注意力核函数；我们以数值实验验证了该恒等式成立。
原文摘要 · Abstract (English)
TriAttention is a recent method for shrinking the KV cache of long-reasoning LLMs: it scores each cached key by how much attention it is likely to receive and evicts the lowest-scoring ones. Because a key does not know how far away its future queries will sit, the score is averaged over a ladder of 17 possible future distances. We point out that this average is free: the future distance enters the score only through the position-dependent rotation, so the whole 17-fold average collapses--exactly, by a one-line algebraic identity--into a single per-band weight that is computed once, offline. Scoring a key then costs one evaluation instead of seventeen, with no change to which keys get pruned. The saving is modest and lives entirely in TriAttention's pruning-score computation, not in the attention kernel; we present it as a small, exact complement to their method, and we confirm the identity numerically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。