提出CRePE方法,用二维上下文提升剪枝精度,配合新搜索策略大幅缩短调参时间。
CRePE: Convolution-aware Relative Importance in Post-training Pruning with Efficient Search

- 引入二维局部邻域信息与自适应系数,改进重要性评分机制
- 在多种模型和稀疏度下超越现有剪枝方法,最高保留95%原模型性能
- 新搜索方法仅需20分钟,且参数可跨模型通用,适合部署优化
将大型语言模型(LLM)投入实际应用面临巨大的内存和计算开销。后训练剪枝(PTP)通过移除权重而不需额外训练来降低这些开销。现有方法中,RIA利用行/列归一化的重要性分数,达到当前最佳精度。但RIA仅考虑一维十字形方向信息,且对行与列贡献赋等权。本文提出CRePE,将二维局部邻域上下文与自适应系数融入相对重要性评分。CRePE在多种模型和稀疏度设置下均优于现有方法。然而,基于困惑度(PPL)的爬山法寻找最优自适应系数需大量PPL评估,耗时约11小时。为此,我们提出PHO(基于代理的超参数优化),消除重复PPL测量,将搜索时间降至约20分钟。此外,PHO在一个模型上找到的最优配置能良好迁移到其他模型,表现出强泛化能力。最后,验证了CRePE可与通道重排、非均匀稀疏分配及再剪枝方法正交结合。
原文摘要 · Abstract (English)
Deploying Large Language Models (LLMs) in practice incurs substantial memory and computational costs. Post-training pruning (PTP) is an effective approach to reducing these costs by removing weights without additional training. Among existing methods, RIA introduces relative importance scores normalized by row and column sums, achieving state-of-the-art accuracy. However, RIA considers only 1D cross-shaped (row/column) directional information and assigns equal weight to row and column contributions. In this paper, we propose \textbf{CRePE}, which incorporates 2D local neighborhood context and adaptive coefficients into Relative Importance scoring. CRePE consistently outperforms existing PTP methods across diverse models and sparsity settings. However, identifying optimal adaptive coefficients via perplexity (PPL)-based hill climbing requires numerous PPL evaluations and approximately 11 hours of search time. To address this, we propose \textbf{PHO} (Proxy-based Hyperparameter Optimization), which eliminates the need for repeated PPL measurements and reduces the search time to approximately 20 minutes. Furthermore, the optimal hyperparameter configuration found by PHO on one model transfers well to other models, demonstrating strong generalization. Finally, we verify that CRePE can be orthogonally combined with existing techniques including Channel Permutation, non-uniform sparsity allocation, and re-pruning methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。