不丢弃任何令牌，实现20倍的缓存压缩，保持高精度。
AnchorKV: Anchor-Residual KV Cache Compression
- 用少量精确锚点表示所有内容，其余通过最相似锚点近似。
- 700亿参数模型上保留99%原始性能，压缩比达20倍。
- 适合长文本生成、高吞吐推理等对内存敏感场景。
键值（KV）缓存是长上下文大模型推理中的主要内存瓶颈。现有方法从两端入手：淘汰策略永久丢弃标记，一旦被丢弃的标记后续变得重要，性能就会下降；量化方法虽保留所有标记但压缩率有限。我们提出AnchorKV，一种压缩方案，在不丢弃任一标记的前提下将缓存缩小20倍。AnchorKV使用一组小规模精确锚点存储缓存，其余每个标记均通过其最相似锚点表达，并仅对影响模型输出最大的近似项进行微调。该方法在不同模型和数据集上均保持一致高精度，在70B规模模型上仍能维持99%的全缓存得分，同时使整个上下文仅需极低内存开销。
原文摘要 · Abstract (English)
The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by $20\times$ without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。