对比多种压缩技术在长文本任务中的表现，发现压缩率不能决定实际效果。
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
- 按任务类型和上下文长度系统评测量化、剪枝、合并等压缩方法
- 压缩率高不一定速度快，不同方法在不同任务上表现差异大
- 适合做长文本服务部署优化，尤其需根据任务选方案
大型语言模型服务在长上下文场景下日益受制于键值缓存（KV-cache）膨胀问题。现有压缩技术难以横向比较，因其评估基于不同模型、任务、预算及服务栈。本文构建了一个面向工作负载的基准测试，涵盖量化、剪枝、合并三类代表性优化方法（KIVI、TurboQuant、SnapKV、CaM），在Llama-3.1-8B-Instruct与Mistral-7B-Instruct-v0.3上，针对LongBench风格的多文档问答、单文档问答、少样本学习和摘要任务进行评估。指标包括任务质量、平均输出吞吐量、平均首字延迟及实际压缩比，覆盖不同上下文长度区间。结果表明：仅看压缩比无法预测端到端性能；KIVI4在跨模型上质量最稳定，SnapKV在长上下文吞吐表现最优，CaM在部分问答任务中提升显著，但对任务敏感且压缩比波动大。研究强调应根据具体任务选择适配的缓存优化策略，为长上下文服务部署提供指导。
原文摘要 · Abstract (English)
Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. This paper presents a workload-aware benchmark of representative KV-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。