提出GradMAP方法,实现快速且高效的大模型层剪枝。
GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensation
- 基于梯度幅值设计全局层重要性度量,单次反向传播即可完成评估。
- 通过投影补偿矩阵修复剪枝导致的层输出偏移,性能下降显著减少。
- 相比已有方法提速4倍以上,适合追求部署效率的模型压缩场景。
大语言模型虽具强大推理能力,但高计算成本限制其实际部署。现有研究发现模型层存在显著冗余,层剪枝成为热点。当前工作主要关注层重要性度量与剪枝后性能恢复,但难以兼顾效率与效果。本文提出GradMAP,包含两个阶段:第一阶段引入基于梯度幅值的新度量,仅需每轮剪枝一次反向传播,极大提升效率;第二阶段分析剪枝后均值偏移最大的层,并引入简单有效的投影补偿矩阵,在一步内修正该偏移,有效缓解性能下降。大量实验表明,GradMAP在剪枝速度上平均提升4倍,同时性能优于现有方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit strong reasoning abilities, but their high computational costs limit their practical deployment. Recent studies reveal significant redundancy in LLMs layers, making layer pruning an active research topic. Layer pruning research primarily focuses on two aspects: measuring layer importance and recovering performance after pruning. Unfortunately, the present works fail to simultaneously maintain pruning performance and efficiency. In this study, we propose GradMAP, a faster layer pruning method with \textbf{Grad}ient \textbf{M}etric \textbf{A}nd \textbf{P}rojection compensation, which consists of two stages. In the first stage, we introduce a novel metric based on gradient magnitudes, enabling a global assessment of layer importance. Note that, it requires only a single backward propagation step per pruning decision, substantially enhancing pruning efficiency. In the second stage, we first analyze the layers with the largest mean shift resulting from pruning, and then incorporate a simple yet effective projection compensation matrix to correct this drift in one step. In this way, the degradation of model performance caused by layer pruning is effectively alleviated. Extensive experiments show that GradMAP outperforms previous layer pruning methods in both pruning speed (achieving an average $4\times$ speedup) and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。