通过成对剪枝压缩大模型键值缓存，保持位置编码有效性。
RAP: KV-Cache Compression via RoPE-Aligned Pruning
- 按旋转位置编码的配对结构进行剪枝，避免破坏位置语义。
- 在30%缓存压缩下准确率几乎不变，优于传统通道剪枝。
- 适合需要长上下文推理的模型部署，可与量化等方法结合使用。
大语言模型在长上下文推理中受键值（KV）缓存内存与计算瓶颈制约。结构化剪枝可通过删除投影权重中意义较小的通道来缩小输出维度，从而压缩缓存。然而，现代LLM在QK投影后使用旋转位置编码（RoPE），其以成对方式旋转特征维度。若直接剪除单个通道会破坏这些配对，导致位置语义失真，使模型不可用。本文提出罗佩对齐剪枝（RAP），将剪枝粒度限定为完整的罗佩对，而非单个通道，以保持旋转一致性。在3B至14B规模的Llama、Mistral和Qwen模型上评估显示，RAP在30%缓存压缩（保留率ρ=0.7）下仍能保持准确率，显著优于无视罗佩结构的通道剪枝，在更低注意力开销下接近最强低秩方法，并可与量化等正交方法兼容。
原文摘要 · Abstract (English)
Long-context inference in large language models (LLMs) is bottlenecked by the memory and compute of the key-value (KV) cache. Structured pruning is a direct way to shrink it: dropping the least useful channels of the W_k, W_v projection weights to reduce the output KV dimensions. However, modern LLMs apply Rotary Position Embedding (RoPE) after the QK projections, which rotates feature dimensions in pairs. Therefore, removing individual channels breaks these pairs, corrupting RoPE's positional semantics and rendering the pruned model unusable. We propose RoPE-Aligned Pruning (RAP), which constrains the pruning granularity to RoPE-aligned pairs rather than individual channels: removing whole pairs to keep the rotation intact. Our evaluation across Llama, Mistral, and Qwen models from 3B to 14B shows that RAP preserves accuracy at 30% KV compression (retain ratio \r{ho} = 0.7), far outperforms RoPE-blind channel pruning, stays near the strongest low-rank method at lower attention cost, and composes with orthogonal methods such as quantization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。