根据输入自适应选择压缩方案，让大模型推理更省内存、更快响应。
MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
- 基于轻量预测模型动态选最优压缩配置
- 在不同约束下成功率达70%以上，最高提升13.5个百分点
- 适合资源受限场景下的长文本推理应用
键值缓存（KV cache）压缩能有效降低大语言模型推理的内存开销，尤其适用于长上下文任务。然而现有方法在精度、延迟和峰值内存占用间存在不同权衡，单一固定配置难以适应不同提示和资源限制。我们提出MetaKV，一个自适应框架，可根据用户指定的延迟和峰值内存预算，为每个输入提示选择最优的压缩配置。MetaKV利用轻量级预测模型估算每种候选配置的端到端延迟、峰值内存及正确回答概率，从而选出在满足约束前提下最优的配置。我们在数学、科学、常识推理和阅读理解四个数据集上，对三种代表性压缩方法（KVQuant、H$_2$O、RocketKV）及未压缩的FP16配置共十种组合进行了评估。在广泛延迟与峰值内存约束下，MetaKV始终优于最佳静态配置，平均提升受约束成功率（CSR）约0.07，最高达0.135。结果证明，针对提示和约束动态调整压缩策略具有显著优势。代码已开源。
原文摘要 · Abstract (English)
Key--value (KV) cache compression is an effective way to reduce the memory overhead of large language model (LLM) inference, particularly for long-context workloads. However, existing compression methods make different trade-offs among accuracy, inference latency, and peak KV cache memory utilization, making a single fixed configuration unsuitable across different prompts and resource constraints. We introduce MetaKV, an adaptive framework that selects a KV cache compression configuration for each input prompt based on user-specified latency and peak memory budgets. MetaKV uses lightweight prediction models to estimate the end-to-end latency, peak memory, and probability of a correct response for each candidate configuration, and selects the configuration that best satisfies the latency-memory constraints while preserving accuracy. We evaluate MetaKV across ten configurations from three representative KV cache compression methods, KVQuant, H$_2$O, and RocketKV, together with an uncompressed FP16 configuration, on four datasets covering mathematics, science, commonsense reasoning, and reading comprehension. Across a wide range of latency and peak memory constraints, MetaKV consistently outperforms the best static configuration, improving constrained success rate (CSR), the fraction of prompts answered correctly while satisfying both constraints, by approximately 0.07 on average and up to 0.135. These results demonstrate the benefit of adapting KV cache compression to individual prompts and latency-memory constraints. Code is available at https://github.com/MichaelWang0505/MetaKV.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。