按重要性分层量化KV缓存，提升大模型推理传输效率
SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving
- 为不同重要性的提示词分配不同精度：高、中、低分别用FP16、INT8、INT4
- 在相同传输预算下，比现有方法降低困惑度25%以上，且对多种模型稳定有效
- 自适应探测机制可动态适配模型容忍度，适合部署时需高效传输的场景
预填充-解码（PD）解耦使提示处理与生成分离，但也导致键值（KV）缓存成为网络负载。现有方法多为二值选择：部分令牌全精度传输，其余不传。本文提出SpectrumKV，为每个令牌分配不同精度：高重要性令牌用FP16，中等用INT8，低重要性在模型可接受时用INT4。关键挑战是INT4容错能力依赖模型：Qwen2.5-7B在INT4下崩溃，而Mistral-7B和Gemma-2-9B仍稳定。SpectrumKV引入轻量级运行时探测：三轮激进的NIAH测试，通过则采用三精度策略，失败则退至双精度。在Qwen2.5-7B-Instruct、Mistral-7B-Instruct-v0.3、Gemma-2-9B-it上，均实现相同传输预算下的质量提升。在WikiText-2数据集上，50%归一化KV预算下，困惑度变化分别为+1.97%、-0.06%、-0.44%，优于PDTrim的+25.85%、+22.07%、+35.63%。在4096令牌的NIAH检索任务中，于b=0.3预算下Qwen达到52.6%准确率（PDTrim为26.3%），在b=0.5时达100%；Mistral与Gemma保持检索性能。端到端GPU时延显示，在b=0.5时首令牌传输时间（TTFT）减少50%-62%。结果表明，PD KV传输应视为精度分配问题，而不仅是令牌剪枝。
原文摘要 · Abstract (English)
Prefill-decode (PD) disaggregation decouples prompt processing from token generation, but it also turns the key-value (KV) cache into a network payload. Existing PD-side KV reduction methods are mostly binary: selected tokens are transmitted at full precision and the rest are not transmitted. This paper argues that binary selection leaves a useful design space unused. SpectrumKV assigns a precision level to each token instead: attention sinks and other high-importance tokens are protected at FP16, medium-importance tokens are sent at INT8, and low-importance tokens are sent at INT4 when the model can tolerate it. The main practical complication is that INT4 tolerance is model-dependent. Qwen2.5-7B catastrophically fails under INT4 KV quantization, while Mistral-7B and Gemma-2-9B remain stable. SpectrumKV therefore runs a lightweight deployment-time probe: three aggressive NIAH trials under a 3-tier policy. Models that pass use FP16+INT8+INT4; models that fail fall back to FP16+INT8. Across Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Gemma-2-9B-it, SpectrumKV improves quality at the same transfer budget. At a 50% normalized KV budget on WikiText-2, SpectrumKV changes perplexity by +1.97%,-0.06%, and-0.44%, respectively, compared with PDTrim's +25.85%, +22.07%, and +35.63%. On NIAH retrieval at 4096 tokens, the adaptive policy reaches 52.6% on Qwen at the aggressive b=0.3 budget versus 26.3% for PDTrim, and reaches 100% by b=0.5; Mistral and Gemma preserve retrieval under the 3-tier policy. End-to-end GPU timing of the transfer path shows 50-62% TTFT reductions at b=0.5. These results suggest that PD KV transfer should be treated as a precision-allocation problem, not only as token pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。