用注意力匹配实现快速低损的长文本压缩
Fast KV Compaction via Attention Matching
- 通过注意力匹配构建紧凑的键值对，保持注意力分布
- 在秒级内完成50倍压缩，性能损失极小
- 适合需要高效处理长上下文的部署场景
将语言模型扩展至长上下文时，关键-值（KV）缓存大小常成为瓶颈。现有方法多通过分词空间的摘要进行压缩，但易造成严重信息损失，影响下游性能。近期工作Cartridges证明可在潜在空间训练高度紧凑的KV缓存以接近完整上下文表现，但需耗时昂贵的端到端优化。本文提出基于注意力匹配的快速潜空间压缩方法，通过构造紧凑的键值对来重现注意力输出并保持每头注意力质量。该框架可分解为若干简单子问题，部分具有高效闭式解。在此基础上，我们设计了一类新方法，在压缩时间与质量的权衡上显著提升性能，某些数据集上可在秒级实现50倍压缩，且性能损失微乎其微。
原文摘要 · Abstract (English)
Scaling language models to long contexts is often bottlenecked by the size of the key-value (KV) cache. In deployed settings, long contexts are typically managed through compaction in token space via summarization. However, summarization can be highly lossy, substantially harming downstream performance. Recent work on Cartridges has shown that it is possible to train highly compact KV caches in latent space that closely match full-context performance, but at the cost of slow and expensive end-to-end optimization. This work describes an approach for fast context compaction in latent space through Attention Matching, which constructs compact keys and values to reproduce attention outputs and preserve attention mass at a per-KV-head level. We show that this formulation naturally decomposes into simple subproblems, some of which admit efficient closed-form solutions. Within this framework, we develop a family of methods that significantly push the Pareto frontier of compaction time versus quality, achieving up to 50x compaction in seconds on some datasets with little quality loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。