4-bit KV缓存压缩提升长上下文代理的推理效率
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
- 采用4比特量化与旋转编码，优化键值缓存存储
- 在长上下文代理任务中，首字延迟降低3.47倍，吞吐提升1.63倍
- 适配AMD GPU，支持FP4近似计算与硬件级加速
面向长上下文代理任务，传统键值（KV）缓存面临巨大压力：长前缀被多次重用，高并发要求系统保持GPU高效利用。本文研究4比特KV缓存压缩，以TurboQuant的旋转机制和码本量化为质量基准，vLLM的FP8缓存为部署基准。提出三项贡献：首先，建立多轮代理任务下的4比特缓存评估框架，需联合衡量任务质量、缓存驻留率与服务吞吐；其次，设计实用优化策略，包括非对称处理K/V、Walsh-Hadamard旋转、移除QJL、块级变体等，提升4比特路径鲁棒性；最后，针对AMD GPU实现服务优化，包括定制解码-注意力内核，并提出UltraQuant方案——使用FP8查询、FP4 KV张量、UE8M0分组尺度及CDNA4原生缩放MFMA支持。在长上下文多轮代理负载下，UltraQuant在缓存压力大的后期回合将P50首字延迟降低3.47倍（全周期2.3倍），输出吞吐相较FP8基线提升1.63倍。
原文摘要 · Abstract (English)
Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system can keep GPUs utilized. We study 4-bit KV-cache compression for this setting, using TurboQuant-style rotation and codebook quantization as a quality anchor and vLLM FP8 KV caching as the deployment anchor. We report three contributions. First, we frame 4-bit KV caching around multi-round agent workloads where task quality, cache residency, and serving throughput must be measured jointly. Second, we describe the practical design choices needed to make the 4-bit path robust, including asymmetric K/V treatment, Walsh-Hadamard rotation, QJL removal, and block-scale variants. Third, we present serving optimizations on AMD GPUs, including optimized decode-attention kernels and UltraQuant, an FP4 approximation path that uses FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA support on CDNA4. On a long-context, multi-turn agentic workload, UltraQuant cuts P50 time-to-first-token by 3.47x in the cache-pressured late rounds (2.3x across all rounds) and raises output throughput by 1.63x over the FP8 KV baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。