SQuat通过正交约束降低大模型缓存量化误差，提升推理速度与内存效率。
SQuat: Subspace-orthogonal KV Cache Quantization
- 构建查询张量子空间，确保量化误差与关键信息正交。
- 内存峰值降低2.17至2.82倍，吞吐提升2.45至3.60倍。
- 无需微调或校准数据，理论基础扎实，适合部署优化场景。
键值（KV）缓存通过存储先前生成标记的键值张量加速大语言模型解码，减少冗余计算但增加内存开销。现有方法将键值张量压缩为低位表示以缓解此问题；然而，随着生成标记增多，量化误差可能累积，导致不良输出。本文提出SQuat（子空间正交键值缓存量化），首先利用查询张量构建子空间以捕获关键任务信息。在键张量量化过程中，强制重构后的键与原键的差值与该子空间正交，从而最小化量化误差对注意力机制输出的影响。SQuat无需模型微调，无需额外校准数据集进行离线学习，并基于自洽的理论框架。数值实验表明，该方法使峰值内存减少2.17至2.82倍，吞吐提升2.45至3.60倍，且在基准测试中表现优于现有算法。
原文摘要 · Abstract (English)
The key-value (KV) cache accelerates LLMs decoding by storing KV tensors from previously generated tokens. It reduces redundant computation at the cost of increased memory usage. To mitigate this overhead, existing approaches compress KV tensors into lower-bit representations; however, quantization errors can accumulate as more tokens are generated, potentially resulting in undesired outputs. In this paper, we introduce SQuat (Subspace-orthogonal KV cache quantization). It first constructs a subspace spanned by query tensors to capture the most critical task-related information. During key tensor quantization, it enforces that the difference between the (de)quantized and original keys remains orthogonal to this subspace, minimizing the impact of quantization errors on the attention mechanism's outputs. SQuat requires no model fine-tuning, no additional calibration dataset for offline learning, and is grounded in a theoretical framework we develop. Through numerical experiments, we show that our method reduces peak memory by 2.17 to 2.82, improves throughput by 2.45 to 3.60, and achieves more favorable benchmark scores than existing KV cache quantization algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。