通过扰动对比迭代剪枝，让大模型瘦身20%仍保持85%以上精度
PIP: Perturbation-based Iterative Pruning for Large Language Models
- 基于原始与扰动两个视角的梯度差异，逐轮剪掉难区分的参数
- 剪枝后参数减少约20%，多数任务精度保留超85%，部分仅降5%
- 适合资源受限场景下部署大模型，优于现有主流剪枝方法
大型语言模型（LLMs）参数量迅速增长至数十亿甚至上万亿，给资源受限环境下的实际部署带来挑战。为此，我们提出一种基于扰动的迭代剪枝方法PIP（Perturbation-based Iterative Pruning），采用双视角结构化剪枝策略，融合原始输入与扰动输入下的信息。通过计算梯度差异，PIP 迭代识别并剪除难以区分两视图的参数。实验表明，PIP 可将参数量减少约20%，在多个基准测试中保持超过85%的原始模型精度；在某些情况下，剪枝后模型性能仅比原模型低5%以内，证明其能有效保留关键模型能力。此外，PIP 持续优于现有最先进（SOTA）的结构化剪枝方法，成为受限环境下优化大模型的领先技术。
原文摘要 · Abstract (English)
The rapid increase in the parameter counts of Large Language Models (LLMs), which often reach into the billions or even trillions, presents significant challenges for their practical deployment, particularly in resource-constrained environments. To address this issue, we propose PIP (Perturbation-based Iterative Pruning), a novel double-view structured pruning method to optimize LLMs, which combines information from two different views: the unperturbed view and the perturbed view. With the calculation of gradient differences, PIP iteratively prunes those that struggle to distinguish between these two views. Our experiments show that PIP reduces the parameter count by approximately 20% while retaining over 85% of the original model's accuracy across varied benchmarks. In some cases, the performance of the pruned model is within 5% of the unpruned version, demonstrating PIP's ability to preserve key aspects of model effectiveness. Moreover, PIP consistently outperforms existing state-of-the-art (SOTA) structured pruning methods, establishing it as a leading technique for optimizing LLMs in constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。