用170万参数模型预测注意力缓存重要性，结果不如简单规则。
On the Limits of Learned Importance Scoring for KV Cache Compression
- 用键值缓存表示预测令牌重要性，不依赖查询信息。
- 多种基线方法（如保留前4个+后N个）表现更优。
- 位置信息和预填充注意力已足够，复杂模型无优势。
我们研究通过推测重要性预测（SIP）实现的可学习键值缓存压缩，SIP是一个170万参数的非查询感知评分器，仅从键值表示中预测令牌重要性。尽管架构复杂（多时域前瞻、跨注意力），SIP在5个随机种子、4种保留率和3项任务下均未超越简单基线，包括随机选择。关键发现：(1) 基于位置的启发式策略（保留前4个+最后N个令牌）表现与或优于可学习方法；(2) 预填充注意力提供的信号与复杂可学习评分器相当；(3) 键值表示中超出位置和预填充注意力的边际信息对重要性预测作用有限。我们假设未来查询与生成轨迹间的循环依赖是造成此难题的原因。
原文摘要 · Abstract (English)
We investigate learned KV cache compression through Speculative Importance Prediction (SIP), a 1.7M parameter non-query-aware scorer that predicts token importance from KV representations alone. Despite architectural sophistication (multi-horizon lookahead, cross-attention), SIP does not outperform simple baselines, including random selection, across 5 seeds, 4 retention levels, and 3 tasks. Key findings: (1) position-based heuristics (keep first 4 + last N tokens) match or exceed learned approaches; (2) prefill attention provides equivalent signal to complex learned scorers; (3) marginal information in KV representations beyond position and prefill attention appears limited for importance prediction. We hypothesize that circular dependence between future queries and generation trajectories contributes to this difficulty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。