提出新方法恢复低比特缓存的分布，让长文本生成更准更快
High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
- 通过检测量化后逻辑值的局部分布风险，仅修复关键候选分布
- 一比特量化下准确率从47.8%提升至83.2%，分布漂移降为0.14
- 适用于大模型长序列推理，内存占用低且解码开销小
长上下文大语言模型推理依赖KV缓存避免重复注意力计算，但带来高内存和带宽开销。低比特KV缓存量化可降低开销，却严重损害性能：在RULER评测下，一比特量化使Llama-3.1-8B准确率从84.2%降至47.8%。我们发现根本原因并非逻辑值绝对误差，而是顶层逻辑值分布的结构化局部错序。为此提出局部分布恢复技术，通过量化逻辑特征检测高风险步骤，仅在词元选择前恢复选定的top-K候选分布。实现DGAP，包含高效的风险检测器与修正器。实验显示，在Llama-3.1-8B上，DGAP将K1V1 RULER准确率从47.8%恢复至83.2%，分布漂移由0.38降至0.14；在Llama、Mistral和Qwen系列模型上，均保持低比特缓存占用并仅有适度解码开销。
原文摘要 · Abstract (English)
Long-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads. Low-bit KV-cache quantization reduces this cost, yet it severely degrade quality; particularly, one-bit quantization reduces accuracy from 84.2% to 47.8% on Llama-3.1-8B under RULER. Rather than common beliefs that absolute error of logits, we find that the root cause is structured local misranking, where the distribution of logits in top-K region is drifted. We thereby propose local distribution restoration, a new technique that detects steps with high local distribution risk from quantized-logit features and restores only the selected top-K candidate distribution before token selection. We implement DGAP to achieve local distribution restoration, with efficient risk detcetors and correctors. Expeirments show that on Llama-3.1-8B, DGAP recovers K1V1 RULER accuracy from 47.8% to 83.2% and reduces distribution drift from 0.38 to 0.14; across Llama, Mistral, and Qwen models, it preserves the persistent low-bit KV-cache footprint with modest decode overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。