通过分析全局困惑度，高效剪枝大模型的键值缓存，提升推理速度与内存效率。
KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models
- 基于全局困惑度评估每层重要性，动态裁剪非关键键值通道
- 推理内存减少50%，吞吐量提升超35%，性能损失极小
- 仅需两小时低秩微调即可恢复性能，适合部署优化
大语言模型推理过程中，键值（KV）缓存成为主要瓶颈。深度剪枝虽能加速推理，但需长达两周的恢复训练；宽度剪枝性能保持较好，但提速有限。为此，我们提出KVPruner，通过全局困惑度分析确定每层重要性比例，并提供多种策略剪除层内非必要KV通道。相比原模型，该方法将运行时内存使用降低50%，吞吐量提升超过35%。此外，仅需在小数据集上进行两小时的LoRA微调，即可恢复大部分性能。
原文摘要 · Abstract (English)
The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。