动态调整每层剪枝率，让大模型在高稀疏下仍保持性能。
DLP: Dynamic Layerwise Pruning in Large Language Models
- 根据权重和输入激活动态评估各层重要性，自适应分配剪枝率。
- 70%稀疏度下，LLaMA2-7B困惑度降低7.79，准确率提升2.7%。
- 兼容多种压缩技术，适合需要高效推理的模型部署场景。
剪枝已被广泛用于降低大语言模型（LLM）的参数量并提升推理效率。主流剪枝方法通常采用均匀分层策略，导致高稀疏度下性能严重下降。鉴于不同层对模型贡献不一，近期研究转向非均匀分层剪枝，但多依赖预设值，难以达到最优效果。为此，我们提出一种新方法——动态分层剪枝（DLP），通过融合模型权重与输入激活信息，自适应评估每层相对重要性，并据此分配剪枝率。实验表明，DLP在多个大模型上均能有效保持高稀疏度下的性能表现。具体而言，在70%稀疏度下，相较于现有最优方法，LLaMA2-7B的困惑度降低7.79，平均准确率提升2.7%。此外，DLP可兼容多种现有压缩技术，并可无缝集成至参数高效微调（PEFT）中。代码已开源：https://github.com/ironartisan/DLP，以促进后续研究。
原文摘要 · Abstract (English)
Pruning has recently been widely adopted to reduce the parameter scale and improve the inference efficiency of Large Language Models (LLMs). Mainstream pruning techniques often rely on uniform layerwise pruning strategies, which can lead to severe performance degradation at high sparsity levels. Recognizing the varying contributions of different layers in LLMs, recent studies have shifted their focus toward non-uniform layerwise pruning. However, these approaches often rely on pre-defined values, which can result in suboptimal performance. To overcome these limitations, we propose a novel method called Dynamic Layerwise Pruning (DLP). This approach adaptively determines the relative importance of each layer by integrating model weights with input activation information, assigning pruning rates accordingly. Experimental results show that DLP effectively preserves model performance at high sparsity levels across multiple LLMs. Specifically, at 70% sparsity, DLP reduces the perplexity of LLaMA2-7B by 7.79 and improves the average accuracy by 2.7% compared to state-of-the-art methods. Moreover, DLP is compatible with various existing LLM compression techniques and can be seamlessly integrated into Parameter-Efficient Fine-Tuning (PEFT). We release the code at https://github.com/ironartisan/DLP to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。