通过频率引导的深度共享，实现长文本推理中键值缓存的高效压缩。
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
- 将相邻层的键值状态分解为低频共享成分与高频残差，动态分配缓存模式。
- 在32k上下文窗口下，准确率接近完整缓存，吞吐达70.4词/秒。
- 适合需要长上下文推理的场景，如问答、代码生成和摘要任务。
长上下文大模型推理受限于键值（KV）缓存的内存与带宽开销，而激进压缩可能破坏多步推理所需的层间特征。本文提出FreqDepthKV，一种推理时的缓存压缩方法，将相邻层的KV状态分解为共享的低频深度成分与稀疏的高频残差。轻量级在线探测器根据注意力头对重建敏感注意力得分的贡献，动态分配其至共享深度、残差深度或精确缓存模式，无需重训练即可适配不同提示结构。在长上下文问答、针状检索、摘要与代码生成等基准测试中，该方法在显著更小的缓存预算下保持任务准确率。当预填充窗口为32k token时，达到58.3 Exact Match、63.0 F1、32.5 ROUGE-L与48.1 pass@1，接近全缓存表现，优于已有压缩方法。同时解码吞吐提升至70.4 tokens/s，TTFT降至2.06秒，峰值KV内存降低至6.2 GB，实现3.9倍有效压缩比。
原文摘要 · Abstract (English)
Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning. We introduce FreqDepthKV, an inference-time cache compression method that factorizes adjacent-layer KV states into shared low-frequency depth components and sparse high-frequency residuals. A lightweight online probe assigns attention heads to shared-depth, residual-depth, or exact cache modes according to their contribution to reconstruction-sensitive attention logits, allowing the compression policy to adapt to prompt structure without retraining. Across long-context question answering, needle retrieval, summarization, and code generation benchmarks, FreqDepthKV preserves task accuracy under substantially smaller cache budgets. With a 32k-token prefill window, FreqDepthKV reaches 58.3 Exact Match, 63.0 F1, 32.5 ROUGE-L, and 48.1 pass@1, closely matching full KV while outperforming prior compressed-cache methods. It also improves decoding throughput to 70.4 tokens/s, reduces TTFT to 2.06 seconds, and lowers peak KV memory to 6.2 GB, achieving a 3.9x effective compression ratio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。