按层非均匀剪枝，让大模型更省力还更好用
Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models
- 用博弈论的谢泼德值衡量每层重要性，分配不同剪枝比例
- 在70%剪枝率下，困惑度降低18%-19%，优于传统方法
- 设计高效近似算法，计算成本大幅下降，适合实际部署
大语言模型剪枝是降低模型规模与计算复杂度的可行方案，但传统逐层均匀剪枝因未考虑各Transformer层重要性差异，导致性能不佳。为此，本文提出基于谢泼德值的非均匀剪枝（SV-NUP）方法，量化每层对整体性能的贡献，实现差异化剪枝预算分配，保留关键参数。为提升效率，设计滑动窗口近似方法，显著降低精确计算谢泼德值的开销。在LLaMA-v1、LLaMA-v2和OPT等多款模型上的实验表明，非均匀剪枝能显著提升剪枝后模型性能。尤其在70%剪枝率下，相比SparseGPT，LLaMA-7B和LLaMA-13B的困惑度（PPL）分别降低18.01%和19.55%。
原文摘要 · Abstract (English)
Pruning large language models (LLMs) is a promising solution for reducing model sizes and computational complexity while preserving performance. Traditional layer-wise pruning methods often adopt a uniform sparsity approach across all layers, which leads to suboptimal performance due to the varying significance of individual transformer layers within the model not being accounted for. To this end, we propose the Shapley Value-based Non-Uniform Pruning (SV-NUP) method for LLMs. This approach quantifies the contribution of each transformer layer to the overall model performance, enabling the assignment of tailored pruning budgets to different layers to retain critical parameters. To further improve efficiency, we design the Sliding Window-based Shapley Value approximation method. It substantially reduces computational overhead compared to exact SV calculation methods. Extensive experiments on various LLMs including LLaMA-v1, LLaMA-v2 and OPT demonstrate the effectiveness of the proposed approach. The results reveal that non-uniform pruning significantly enhances the performance of pruned models. Notably, SV-NUP achieves a reduction in perplexity (PPL) of 18.01% and 19.55% on LLaMA-7B and LLaMA-13B, respectively, compared to SparseGPT at 70% sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。