通过联合优化缓存与模型压缩，实现长文本推理的内存大幅降低。
KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
- 构建系统级框架，权衡缓存量化、分块预填充与权重压缩的组合效果。
- 在128k上下文长度下，内存减少68%-78%，准确率损失仅1%-3%。
- 适合边缘部署的长序列推理场景，尤其对资源受限设备有实用价值。
长上下文大语言模型在推理时面临显著的内存瓶颈，因键值（KV）缓存随序列长度线性增长。尽管单独的优化技术如KV缓存量化、分块预填充和模型权重量化已展现出潜力，但它们的协同效应及面向边缘部署的最优配置仍待探索。我们提出KV Pareto，一个系统级框架，全面映射总内存消耗与任务准确率之间的权衡前沿，涵盖三种互补优化技术。框架评估了Qwen、Llama、Mistral等多类模型，结合多种KV量化方案（int2/4/8、混合精度）、粒度（逐标记、逐张量、逐块）以及4比特权重量化（通过AWQ）。结果发现，针对不同模型的帕累托最优配置可在长文本任务中实现68%-78%的总内存减少，且准确率下降仅1%-3%。此外，在Needle-in-a-Haystack、GSM8k和MMLU等基准上验证了该前沿的有效性，并扩展至最高128k的上下文长度，证明联合优化对高效推理的实际必要性。
原文摘要 · Abstract (English)
Long-context Large Language Models (LLMs) face significant memory bottlenecks during inference due to the linear growth of key-value (KV) cache with sequence length. While individual optimization techniques like KV cache quantization, chunked prefill, and model weight quantization have shown promise, their joint effects and optimal configurations for edge deployment remain underexplored. We introduce KV Pareto, a systems-level framework that systematically maps the trade-off frontier between total memory consumption and task accuracy across these three complementary optimization techniques. Our framework evaluates multiple LLM architectures (Qwen, Llama, Mistral) with varying KV quantization schemes (int2/4/8, mixed-precision), granularities (per-token, per-tensor, per-block), and 4-bit weight quantization via AWQ. Our framework identifies model-specific Pareto-optimal configurations that achieve 68-78% total memory reduction with minimal (1-3%) accuracy degradation on long-context tasks. We additionally verify the selected frontiers on additional benchmarks of Needle-in-a-Haystack, GSM8k and MMLU as well as extended context lengths of up to 128k to demonstrate the practical need of joint optimization for efficient LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。