自适应选择校准数据，提升大模型结构化剪枝效果
Sample-aware Adaptive Structured Pruning for Large Language Models
- 根据样本动态选择校准数据，优化剪枝指标
- 20%剪枝率下保持97%原始性能
- 适合需要高效部署大模型的研究者
大语言模型在自然语言处理中表现卓越，但庞大的模型规模和高昂的计算成本限制了其实际部署。结构化剪枝可通过移除冗余参数有效降低资源需求。然而，现有方法使用随机选取的校准数据和固定的单一重要性评估指标，导致剪枝后模型性能下降。本文提出AdaPruner，一种面向大语言模型的样本感知自适应结构化剪枝框架，旨在优化剪枝过程中的校准数据与重要性评估指标。具体而言，AdaPruner通过构建结构化剪枝解空间，并采用贝叶斯优化自适应搜索最优校准数据与重要性指标组合。实验表明，该方法在多个不同剪枝率的大语言模型上均优于现有结构化剪枝方法，展现出良好的适用性与鲁棒性。尤其在20%剪枝率下，剪枝后模型保持了97%的原始性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved outstanding performance in natural language processing, but enormous model sizes and high computational costs limit their practical deployment. Structured pruning can effectively reduce the resource demands for deployment by removing redundant model parameters. However, the randomly selected calibration data and fixed single importance estimation metrics in existing structured pruning methods lead to degraded performance of pruned models. This study introduces AdaPruner, a sample-aware adaptive structured pruning framework for LLMs, aiming to optimize the calibration data and importance estimation metrics in the structured pruning process. Specifically, AdaPruner effectively removes redundant parameters from LLMs by constructing a structured pruning solution space and then employing Bayesian optimization to adaptively search for the optimal calibration data and importance estimation metrics. Experimental results show that the AdaPruner outperforms existing structured pruning methods on a family of LLMs with varying pruning ratios, demonstrating its applicability and robustness. Remarkably, at a 20\% pruning ratio, the model pruned with AdaPruner maintains 97\% of the performance of the unpruned model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。