通过选择性恢复线性，实现大模型注意力缓存的高效压缩。
EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
- 基于RoPE特性，选择性恢复关键维度线性，简化缓存优化。
- 联合低秩压缩键值对，实现75%缓存空间减少，性能几乎无损。
- 适配多规模模型，仅需0.6%数据微调，部署友好。
旋转位置编码（RoPE）使每个注意力头能捕捉序列维度上的多频信息，广泛应用于基础模型。然而，RoPE引入的非线性使基于RoPE的注意力中键状态的缓存优化复杂化。现有缓存压缩方法通常在旋转前存储键状态，并在解码时应用变换，带来额外计算开销。本文提出EliteKV，一种支持可变压缩比的RoPE模型灵活修改框架。EliteKV首先使用RoPElite识别各注意力头的内在频率偏好，选择性地恢复部分键维度的线性，以简化注意力计算中的缓存优化。在此基础上，通过键与值的联合低秩压缩，实现部分缓存共享。实验表明，仅用原训练数据的0.6%进行微调，基于RoPE的模型即可实现75%的缓存大小缩减，且性能损失可忽略。此外，EliteKV在同一系列不同规模的模型上均表现稳定。
原文摘要 · Abstract (English)
Rotary Position Embedding (RoPE) enables each attention head to capture multi-frequency information along the sequence dimension and is widely applied in foundation models. However, the nonlinearity introduced by RoPE complicates optimization of the key state in the Key-Value (KV) cache for RoPE-based attention. Existing KV cache compression methods typically store key state before rotation and apply the transformation during decoding, introducing additional computational overhead. This paper introduces EliteKV, a flexible modification framework for RoPE-based models supporting variable KV cache compression ratios. EliteKV first identifies the intrinsic frequency preference of each head using RoPElite, selectively restoring linearity to certain dimensions of key within attention computation. Building on this, joint low-rank compression of key and value enables partial cache sharing. Experimental results show that with minimal uptraining on only $0.6\%$ of the original training data, RoPE-based models achieve a $75\%$ reduction in KV cache size while preserving performance within a negligible margin. Furthermore, EliteKV consistently performs well across models of different scales within the same family.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。