SlimGPT通过分层剪枝提升大模型效率，1小时内实现接近最优的压缩效果。
SlimGPT: Layer-wise Structured Pruning for Large Language Models
- 基于批量贪心剪枝，结合分组分解与动态分组，加速结构化剪枝。
- 在LLaMA上实现比现有方法更优的性能，剪枝后仍保持高准确率。
- 提出非均匀剪枝策略，缓解分层剪枝中的误差累积问题，适合资源受限场景。
大语言模型（LLMs）因其在多个领域的卓越能力而备受关注，但其庞大的参数规模给实际部署带来挑战。结构化剪枝是平衡模型性能与效率的有效方法，但在计算资源受限下恢复性能仍是主要难题。为此，我们提出一种基于最优脑外科（Optimal Brain Surgeon）框架的低成本、快速结构化剪枝方法SlimGPT。提出批量化贪心剪枝（Batched Greedy Pruning），通过分组乔里斯基分解提升头级剪枝误差估计精度，并通过动态分组大小优化前馈网络（FFN）的剪枝效率，从而在1小时内实现近似局部最优的剪枝结果。此外，从误差累积角度分析了分层剪枝的局限性，提出增量剪枝比例（Incremental Pruning Ratio）这一非均匀剪枝策略，有效降低性能退化。在LLaMA基准上的实验表明，SlimGPT优于其他方法，达到当前最佳性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。