通过两阶段正则化实现大模型结构化剪枝，无需重训练即可高效压缩。
Two-Stage Regularization-Based Structured Pruning for LLMs
- 先用可学习权重和L1正则逐步削弱冗余层输出
- 再对弱层施加输入输出差异正则，迁移知识到保留层
- 比传统剪枝方法保性能更强，适合部署优化
大型语言模型（LLMs）的部署受限于参数量过大。结构化剪枝成为有前景的解决方案。以往方法直接根据指标移除不重要参数，常导致知识丢失且需大量重训练。为此，我们提出新型剪枝方法TRSP：基于两阶段正则化的结构化剪枝。首先，为每层Transformer输出乘上初始可学习权重，并通过在损失函数中添加其ℓ₁-范数作为正则项，迭代学习这些权重，作为第一阶段正则化。随后，对权重较小的层，对其输出与输入的差异施加额外正则，促使知识向保留层转移，作为第二阶段正则化。TRSP能更好保留知识和模型性能，优于直接删除参数的方法。大量实验表明，TRSP在无需重训练的情况下超越强基准层级剪枝方法，作为层级剪枝法，可显著实现端到端加速，是高效部署大模型的有力方案。代码见https://github.com/fmk345/TRSP。
原文摘要 · Abstract (English)
The deployment of large language models (LLMs) is largely hindered by their large number of parameters. Structural pruning has emerged as a promising solution. Prior structured pruning methods directly remove unimportant parameters based on certain metrics, which often causes knowledge loss and necessitates extensive retraining. To overcome this, we introduce a novel pruning method TRSP: Two-Stage Regularization-Based Structured Pruning for LLMs. Specifically, we multiply the output of each transformer layer by an initial learnable weight and iteratively learn these weights by adding their $\ell_1$-norm as a regularization term to the loss function, serving as the first-stage regularization. Subsequently, we apply additional regularization to the difference between the output and input of layers with smaller weights, encouraging the shift of knowledge to the preserved layers. This serves as the second-stage regularization. TRSP retains more knowledge and better preserves model performance than direct parameter elimination. Through extensive experimentation we show that TRSP outperforms strong layer-wise structured pruning methods without requiring retraining. As a layer-wise pruning method, it delivers notable end-to-end acceleration, making it a promising solution for efficient LLM deployment.Code is available at https://github.com/fmk345/TRSP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。