通过感知质量悬崖的混合精度量化，实现长文本推理6倍存储压缩且无质量损失。
SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
- 基于模型内部评分排序所有token，仅在质量悬崖之上分配两种相邻精度。
- 实测6.0倍压缩率，900次实验下与全精度无统计差异，优于1.5倍内存预算的剪枝方案。
- 发现并利用量化中的质量悬崖现象，适用于追求高内存效率的长上下文LLM部署。
键值（KV）缓存是长上下文大语言模型推理的主要内存瓶颈，其大小随上下文长度线性增长。我们发现，在分数比特网格上的均匀量化并非平滑退化：在预设多种子统计协议下，使用仿射量化器的Llama-3.1-8B-Instruct在2.322比特/值时仍与FP16 KV无统计差异，但在2.0比特时突然崩溃——形成(2.0, 2.322]之间的质量悬崖，该现象在生成时量化和多轮对话中重现，并可迁移至Mistral-7B。这一悬崖重塑了重要性感知的混合精度设计：在悬崖之上，八种模型内重要性指标统计等价，混合精度的优势源于网格插值，实现普通均匀量化无法达到的平均精度。SemKV保留所有token，按模型内得分排序，分配两个高于悬崖的相邻精度，实测达成6.0倍存储压缩，且与完整KV在900次实验、三组种子下无统计差异，优于获得1.5倍更大内存预算的FP16 token pruning。将仿射基替换为失真优化量化器（TurboQuant-MSE）后，所有测试协议下悬崖下移，无损失操作点提升至7.9倍。方法：测量目标部署场景的质量悬崖，然后在上方进行插值。
原文摘要 · Abstract (English)
The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。