动态调整注意力缓存，让大模型长文本处理更省空间、更高效。
DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs
- 按任务需求动态分配每层缓存大小，避免固定压缩浪费
- 仅用1.7%缓存达到全量90%性能，极端压缩下仍超主流方法11%
- 适合需要长文本理解的RAG、摘要等场景，尤其在稀疏信息检索中表现突出
长上下文任务（如RAG和摘要）中，高效的KV缓存管理对大语言模型至关重要。现有压缩方法采用固定模式，忽略任务特性，导致关键信息丢失。我们观察到不同任务在各层激活模式差异显著，因此提出DynamicKV：通过建立全局与逐层缓存预算，在推理过程中动态调整每层保留的令牌数，并周期性更新前序层缓存。该方法仅需1.7%的缓存大小，即可在LongBench上达到约85%的全量缓存性能；在极强压缩（0.9%）下，于Mistral-7B-Instruct-v0.2的Needle-in-a-Haystack测试中超越当前最优方法11%。代码将公开。
原文摘要 · Abstract (English)
Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。