提出轻量级4比特键值缓存压缩方案,兼顾精度与部署效率。
SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving

- 采用逐令牌4比特量化+块对角哈达玛旋转,适配实际服务约束。
- 在多模型上恢复近全精度,相比朴素4比特损失极小。
- 零额外开销集成到分页缓存,适合高并发真实场景使用。
键值缓存(KV-cache)是真实大模型服务中的主要瓶颈,系统需同时支持低延迟小批量请求和高吞吐并发负载。尽管已有多种压缩方法提升离线精度或压缩比,但常违反实际服务约束,如分页内存布局、规律内存访问和融合注意力执行,限制其部署效果。本文识别出在这些约束下仍可行的最小4比特量化方法集合。核心发现是:一种简单设计——逐令牌INT4量化结合块对角哈达玛旋转——始终实现最佳精度-效率权衡。在多个模型和基准测试中,该方法几乎完全恢复了朴素INT4的精度损失,而向量量化、海森量化等复杂方法在考虑服务兼容性后仅带来微弱增益。为实现落地,我们构建了融合旋转-量化内核,直接集成至分页KV-cache布局,引入零可测量端到端开销,在不同并发级别下达到与原生INT4相当的吞吐性能。结果表明,有效的KV-cache压缩本质上是系统协同设计问题:在真实服务约束下,轻量块对角哈达玛旋转是一种可行方案,能实现近乎无损精度且不牺牲服务效率。
原文摘要 · Abstract (English)
KV-cache memory is a major bottleneck in real-world LLM serving, where systems must simultaneously support latency-sensitive small-batch requests and high-throughput concurrent workloads. Although many KV-cache compression methods improve offline accuracy or compression ratio, they often violate practical serving constraints such as paged memory layouts, regular memory access, and fused attention execution, limiting their effectiveness in deployment. In this work, we identify the minimal set of 4-bit KV-cache quantization methods that remain viable under these constraints. Our central finding is that a simple design--token-wise INT4 quantization with block-diagonal Hadamard rotation--consistently achieves the best accuracy-efficiency trade-off. Across multiple models and benchmarks, this approach recovers nearly all of the accuracy lost by naive INT4, while more complex methods such as vector quantization and Hessian-aware quantization provide only marginal additional gains once serving compatibility is taken into account. To make this practical, we implement a fused rotation-quantization kernel that integrates directly into paged KV-cache layouts and introduces zero measurable end-to-end overhead, matching plain INT4 throughput across concurrency levels. Our results show that effective KV-cache compression is fundamentally a systems co-design problem: under real serving constraints, lightweight block-diagonal Hadamard rotation is a viable method that delivers near-lossless accuracy without sacrificing serving efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。