提出新方法评估注意力压缩效果，发现稀疏性不是压缩的唯一依据。
Beyond Sparse Weights: When Is Attention Compressible?
- 基于全局得分差判断需保留多少令牌以维持目标质量
- 提出CertKV压缩器，在多个长文本任务中表现优于主流方法
- 适合需要高效缓存的长序列推理场景
KV-cache压缩常被解释为注意力图中仅有少数大权重。但这一解释不完整：大权重未必包含主要信息量，被忽略值可能相互抵消，且保持注意力输出未必能保留任务性能。本文将问题拆解：全局得分差决定需保留的令牌数量；对某一实际行，被省略值的加权和即为缺失的精确统计量。通过一个受控的检索-聚合模型，解释了截断在何时有益、何时有害。据此提出CertKV，一种无需训练的压缩器，每头保留一个尾部摘要槽，其余按值分布分配。在匹配预算下，CertKV在九个LongBench-v2设置中位列前二，在128K RULER上仍处领先压缩梯队，并在打包的Llama原型中实现十倍缓存预算提升。压缩能力取决于信息质量、具体数值、未来查询及任务需求——而不仅限于看似稀疏的注意力图。
原文摘要 · Abstract (English)
KV-cache compression is often justified by attention maps with a few large weights. This is incomplete: large weights may not contain most of the mass, omitted values can cancel, and preserving the attention output may not preserve the task. We separate these questions. Global score gaps -- not threshold counts -- determine how many tokens are needed to retain a target mass. For a realized row, the weighted sum of omitted values is the exact missing statistic. A controlled retrieval--aggregation model explains when truncation helps and when it hurts. These results motivate CertKV, a training-free compressor that reserves one tail-summary slot per head and allocates the rest by value dispersion. Under matched budgets, CertKV is top-two in seven of nine LongBench-v2 settings, remains in the leading compressed tier on 128K RULER, and realizes a ten-fold cache budget in a packed Llama prototype. Compressibility depends on the mass, values, future queries, and task -- not on a sparse-looking map alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。