通过直接压缩注意力矩阵，显著提升大模型缓存压缩的精度。
KQ-SVD: Compressing the KV Cache with Provable Guarantees on Attention Fidelity
- 直接对注意力矩阵做低秩分解，而非分开压缩键值向量。
- 在LLaMA和Mistral模型上保持更高注意力输出保真度。
- 适用于需要高效推理的长序列大模型部署场景。
键值（KV）缓存是基于Transformer的大语言模型高效推理的核心，用于存储先前计算的向量以加速推理。然而，随着序列长度和批处理大小的增长，缓存成为主要的内存瓶颈。以往的压缩方法通常仅对键向量进行低秩分解，或尝试联合嵌入查询与键向量，但这些方法忽略了注意力机制本质上依赖于两者的内积。本文证明，此类策略在近似注意力矩阵时表现次优。我们提出KQ-SVD，一种简单且计算高效的算法，通过闭式解直接对注意力矩阵进行最优低秩分解。由于瞄准了冗余的真实来源，KQ-SVD在压缩下能更好地保留注意力输出。在LLaMA和Mistral模型上的大量实验表明，该方法始终提供更优的投影质量。
原文摘要 · Abstract (English)
The Key-Value (KV) cache is central to the efficiency of transformer-based large language models (LLMs), storing previously computed vectors to accelerate inference. Yet, as sequence length and batch size grow, the cache becomes a major memory bottleneck. Prior compression methods typically apply low-rank decomposition to keys alone or attempt to jointly embed queries and keys, but both approaches neglect that attention fundamentally depends on their inner products. In this work, we prove that such strategies are suboptimal for approximating the attention matrix. We introduce KQ-SVD, a simple and computationally efficient method that directly performs an optimal low-rank decomposition of the attention matrix via a closed-form solution. By targeting the true source of redundancy, KQ-SVD preserves attention outputs with higher fidelity under compression. Extensive evaluations on LLaMA and Mistral models demonstrate that our approach consistently delivers superior projection quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。