arXiv:2602.08686cs.LGcs.AI2026-02

通过离线编译压缩策略,实现更稳定高效的注意力键值缓存管理。

CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation

  • 离线构建校准数据表,减少在线修正计算量至常数级
  • 在4个模型上跨语料库保持90%相关性,压缩性能达新最佳
  • 适用于高压缩比场景,极端情况下仍可避免内存溢出

仅预填充阶段的键值(KV)压缩会冻结预填充末尾的一组标记并据此解码,不再进行淘汰。因此保留决策不可逆,而现有方法需在线从单一噪声提示中估计其依赖的修正信号——每头可靠性与提示级压缩敏感性。我们提出,这种信号的统计单位有误:这些信号在跨提示间具有远高于提示内信噪比的规律性。本文提出 extsc{CompilerKV},其纠正表通过离线校准语料库生成,将标准观测窗口扫描后的在线修正降至 $O(1)$ 次查找加预算限制。实验发现,编译后的保留表具备可移植的结构先验特性:在四个主干模型上跨不相交语料库排名相关性均值达 Spearman $arρ=0.90$;直接模型间迁移仅损失 $0.4$--$0.8$ 个 LongBench 分数。在512令牌预算下, extsc{CompilerKV} 在所有四个主干模型上均达到压缩性能新最优,平均优于最强预填充基线 $+1.67$ 分(任务引导95%置信区间 $[+1.08,+2.37]$)。在固定 $512/32k$ 缓存比例压力下,其在128k RULER上表现约73分(全键值约79分,SnapKV约38分);在32k NIAH上达0.89分(SnapKV为0.42分);在输入32k时仅保留预填充键值的1.56%,批量16服务仍可行,而全键值已超出内存限制。

原文摘要 · Abstract (English)

Prefill-only KV compression freezes a token subset at the end of prefill and decodes from it without further eviction. The retention decision is therefore irreversible, yet existing methods estimate the corrective signals it relies on, per-head reliability and prompt-level compression sensitivity, online from a single noisy prompt. We argue this is the wrong statistical unit: these signals exhibit far higher cross-prompt regularity than within-prompt signal-to-noise. We introduce \textsc{CompilerKV}, a KV-retention policy whose corrective tables are compiled offline from a calibration corpus, reducing online correction after the standard observation-window scan to $O(1)$ lookups plus a budget clamp. We find that compiled retention tables behave as portable architectural priors: rankings transfer across disjoint corpora on four backbones (mean Spearman $\barρ{=}0.90$), and direct model-to-model table transfer costs only $0.4$--$0.8$ LongBench points on average. At a 512-token budget, \textsc{CompilerKV} attains compressed-SOTA on all four backbones, improving over the strongest prefill-only baseline by $+1.67$ points on average (task-bootstrap 95\% CI $[+1.08,+2.37]$). Pressure regimes amplify the gap: under a fixed $512/32k$ cache ratio, CompilerKV remains the strongest compressed method through 128k RULER ($\sim\!73$ vs.\ FullKV $\sim\!79$, SnapKV $\sim\!38$); on 32k NIAH it reaches $0.89$ vs.\ SnapKV $0.42$; and at 32k input, retaining only $1.56\%$ of the prefill KV, batch-16 serving remains feasible where FullKV is OOM.

KV压缩推理优化缓存管理离线编译

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。