改进大模型剪枝顺序，让剪枝更准更快。
ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning
- 按潜在误差大小重新排序剪枝顺序，先剪影响大的权重。
- 在多个大模型上剪枝后性能超越原始方法，损失更小。
- 适合需要高效部署大模型的开发者和研究者。
剪枝是减少大语言模型参数的有效方法，有助于提升部署与推理效率。经典的一次性剪枝方法如SparseGPT利用二阶梯度（即海塞矩阵），但其预设的从左到右剪枝顺序在权重呈现列模式时表现不佳。本文研究了剪枝顺序对SparseGPT框架的影响，提出ROSE方法：先进行预剪枝识别候选权重，估算列与块级剪枝损失；再分两层重排——块内按列损失降序排列，块间按块损失排序，并引入块损失相对范围作为指标识别列结构层，实现全模型自适应重排。在主流LLM（LLaMA2-7B/13B/70B、LLaMA3-8B、Mistral-7B）上的大量实验表明，ROSE优于原始SparseGPT及其他对比方法。代码已开源：https://github.com/mingluo-su/ROSE。
原文摘要 · Abstract (English)
Pruning is widely recognized as an effective method for reducing the parameters of large language models (LLMs), potentially leading to more efficient deployment and inference. One classic and prominent path of LLM one-shot pruning is to leverage second-order gradients (i.e., Hessian), represented by the pioneering work SparseGPT. However, the predefined left-to-right pruning order in SparseGPT leads to suboptimal performance when the weights exhibit columnar patterns. This paper studies the effect of pruning order under the SparseGPT framework. The analyses lead us to propose ROSE, a reordered SparseGPT method that prioritizes weights with larger potential pruning errors to be pruned earlier. ROSE first performs pre-pruning to identify candidate weights for removal, and estimates both column and block pruning loss. Subsequently, two-level reordering is performed: columns within each block are reordered in descending order of column loss, while blocks are reordered based on block loss. We introduce the relative range of block loss as a metric to identify columnar layers, enabling adaptive reordering across the entire model. Substantial empirical results on prevalent LLMs (LLaMA2-7B/13B/70B, LLaMA3-8B, Mistral-7B) demonstrate that ROSE surpasses the original SparseGPT and other counterpart pruning methods. Our code is available at https://github.com/mingluo-su/ROSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。