arXiv:2605.17757cs.LGcs.AI2026-05被引 3

提出新型2比特键值缓存量化方法,显著提升长文本大模型推理性能。

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

论文配图:OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
图 1 · 摘自论文原文
  • 离线分析注意力协方差结构,设计适配注意力的固定旋转与截断阈值
  • 在32K上下文上保持接近全精度的准确率,误差仅1.42~3.78点
  • 支持现代服务框架,内存降8倍,吞吐最高提升7倍,适合部署

INT2键值缓存量化对长上下文大模型服务极具吸引力,但难以兼顾精度与可部署性。简单旋转如哈达玛变换虽能减少异常值,但在INT2下仍因未对齐下游注意力而性能下降。本文提出OSCAR,一种超低比特键值缓存量化方法:离线估计注意力感知的协方差结构,据此生成固定的旋转与截断阈值,使量化结果与注意力实际消耗的协方差结构对齐。我们不仅提供理论依据,还构建了完整的可部署系统,包含定制INT2注意力核,兼容分页键值缓存与融合内核流水线,可无缝集成至SGLang、vLLM等现代大模型服务框架。在5个任务上评估,使用高达32k token的推理轨迹,Qwen3-4B-Thinking-2507和Qwen3-8B的BF16精度差距分别降至3.78和1.42点,而朴素旋转INT2几乎崩溃。扩展至Qwen3-32B和GLM-4.7(358B参数),仍保持与BF16相当性能。在长达128K的RULER-NIAH数据集上,两个Qwen3模型均保持鲁棒,而朴素旋转方案彻底失效。系统层面,OSCAR将键值缓存内存降低约8倍,相同内存预算下大批次吞吐最高提升7倍,单批次解码速度比BF16快最多3倍,得益于更低的内存带宽开销。

原文摘要 · Abstract (English)

INT2 KV-cache quantization is attractive for long-context LLM serving, but it remains difficult to make both accurate and deployable. Simple rotations such as Hadamard transforms reduce outliers, but still degrade at INT2 because they are not aligned with downstream attention. We propose OSCAR, an Ultra-low-bit KV Cache quantization method that estimates attention-aware covariance structures offline and uses them to derive fixed rotations and clipping thresholds for quantization. In this way, it aligns KV quantization with the covariance structures that attention actually consumes. More importantly, we not only provide theoretical justification but also develop a fully deployable OSCAR system with a custom INT2 attention kernel that remains compatible with paged KV-cache serving and fused kernel pipelines, enabling seamless integration into modern LLM serving frameworks such as SGLang and vLLM. We evaluate our methods on recent reasoning models with reasoning traces of up to 32k tokens across 5 tasks. On Qwen3-4B-Thinking-2507 and Qwen3-8B, OSCAR reduces the BF16 accuracy gap to 3.78 and 1.42 points, respectively, while naive rotation INT2 collapses to nearly zero. We further scale OSCAR to Qwen3-32B and GLM-4.7 (358B params), where it remains effectively on par with BF16. On long context - RULER-NIAH up to 128K, OSCAR remains robust on both Qwen3 models, while naive rotation INT2 collapses. System-wise, OSCAR reduces KV-cache memory by approximately 8x, improves throughput by up to 7x at large batch sizes under the same memory budget, and accelerates batch-size-1 decoding by up to 3x over BF16 due to reduced memory bandwidth overhead.

量化大模型推理优化内存压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。