针对低比特缓存量化误差，提出关注输出的旋转优化方法。
Output-Aware Rotation for INT2 KV-Cache Quantization
- 在完整注意力路径中学习每头正交修正，优化输出端误差
- 在三个模型五项任务上显著优于现有方法，提升长文本检索效果
- 兼容分页缓存格式，推理开销几乎不增加，适合部署
键值（KV）缓存已成为长上下文大语言模型推理中的主要内存与带宽瓶颈，促使超低比特量化愈发重要。然而，现有基于旋转的INT2方法仅在完整注意力读取前优化缓存统计或代理误差，而模型最终受注意力传播误差及输出投影 $W_O$ 影响。为解决此偏差，本文提出 extit{OptR}，一种输出感知的旋转方法，旨在最小化 $W_O$ 后的注意力输出误差。OptR 将 $W_O$ 后的输出误差分解为键和值诱导项，并通过完整的INT2量化与注意力路径学习每头正交修正。此外，采用注意力等效键重参数化，消除大通道偏移而不改变softmax分布。在三个模型和五个推理与编码基准上，OptR 持续优于 QuaRot 与 OSCAR，增强长上下文检索能力，同时保持分页KV缓存格式，推理开销可忽略。
原文摘要 · Abstract (English)
The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important. However, existing rotation-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projection $W_O$. To address this mismatch, we propose \textit{OptR}, an output-aware rotation method that minimizes post-$W_O$ attention-output error. OptR decomposes the post-$W_O$ attention-output error into key- and value-induced terms and learns per-head orthogonal corrections through the full INT2 quantization and attention path. OptR further applies an attention-equivalent key reparameterization to reduce large channel-wise offsets without changing the softmax distribution. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long-context retrieval, while preserving the paged KV-cache format with negligible inference overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。