通过提取跨层关键向量，实现大模型长文本推理的内存与速度双重优化。
xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
- 利用奇异向量对齐发现跨层键值缓存主成分一致，设计联合低秩分解压缩方案。
- 最多实现8倍键值缓存压缩，多轮对话和长文本任务保持准确率不变。
- 支持推理时选择性重建，端到端提速4.23倍，吞吐量提升30%以上。
长上下文大语言模型（LLMs）虽功能强大，但因键值缓存（KV-Cache）占用大量内存。现有跨层共享方法或需昂贵预训练，或依赖逐标记的跨层余弦相似度，实际效果有限。我们通过中心核对齐（CKA）发现，键值缓存的主导奇异向量在各层间高度对齐。受此启发，提出xKV：一种后训练压缩方法，将分组层的键值缓存联合分解至共享低秩子空间，显著降低内存开销。在多种主流LLM上，xKV最多实现8倍键值缓存压缩，同时保持长文本任务与多轮对话中的精度。为进一步提升效率，我们在解码时引入选择性重建（SR）。结合SR后，xKV相比完整注意力基线实现最高4.23倍端到端加速，在相近精度下比知名基线提升30%吞吐量。整体上，xKV为长上下文LLM推理提供了即插即用的内存与延迟优化方案。代码已开源：https://github.com/abdelfattah-lab/xKV。
原文摘要 · Abstract (English)
Long-context Large Language Models (LLMs) enable powerful applications but incur high memory costs due to the key-value states (KV-Cache). Recent studies attempt to share KV-Cache across layers, but these approaches either require expensive pretraining or rely on per-token cross-layer cosine similarity that is often limited in practice. We show, via Centered Kernel Alignment (CKA), that the dominant singular vectors of KV-Cache are well aligned across layers. Motivated by this observation, we propose xKV, a post-training compression method that jointly factorizes grouped-layer KV-Cache into a shared low-rank subspace, substantially reducing KV-Cache memory. Across widely used LLMs, xKV achieves up to 8x KV-Cache compression while preserving accuracy on long-context tasks and in multi-turn settings. To further improve efficiency, we introduce Selective Reconstruction (SR) at decode time. Combined with SR, xKV achieves up to 4.23x end-to-end speedup over the full attention baseline, and surpasses notable baselines with 30% higher throughput under a similar accuracy level. Overall, xKV provides a plug-and-play approach to reduce both memory and latency for long-context LLM inference. Our code is publicly available at: https://github.com/abdelfattah-lab/xKV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。