arXiv:2504.04342cs.CL2025-04中稿 · EMNLP被引 2

提出剪枝定律,可预估大模型剪枝后的性能表现。

Pruning Laws for Large Language Models

  • 发现剪枝后性能与原始性能、剪枝比例间存在可预测的数学关系。
  • 在10个模型上平均外推误差低于7%,能准确量化性能下降。
  • 适用于不同架构和剪枝方法,帮助快速选定安全剪枝比例。

大规模语言模型(LLM)参数和训练数据的增加虽持续提升性能,但导致内存和计算需求急剧上升,使资源受限硬件部署变得不可行。模型剪枝作为一种广泛使用的压缩技术,通过移除冗余参数降低推理成本。然而,其对下游性能的影响难以预测,通常需依赖昂贵的实验测试。为此,我们提出了剪枝定律——一种简单且可解释的缩放关系,将剪枝后模型的性能与原始性能及剪枝比例关联起来。在10个模型(1.3B-30B参数)、一个20B混合专家模型、三种剪枝策略(非结构化、宽度、深度)及八项任务中,剪枝定律实现了高预测精度(平均外推误差小于7%),可靠量化性能退化,并识别出无法恢复的临界剪枝阈值。此外,该函数形式在密集模型与混合专家架构、不同剪枝方法间具有零样本和单样本迁移能力,任务与方法相关的系数变化具有可解释性。这些结果为研究者和从业者提供了系统化框架,用于选择剪枝策略、估计安全剪枝比例,无需反复调参即可高效部署模型。

原文摘要 · Abstract (English)

Scaling up model parameters and training data consistently improves the performance of large language models (LLMs), but at the cost of rapidly growing memory and compute requirements, which makes deployment on resource-limited hardware infeasible. Model pruning, a widely used compression technique, reduces inference costs by removing redundant parameters. However, its impact on downstream performance remains unpredictable and is typically assessed only through costly empirical sweeps. To address this gap, we introduce pruning laws, simple and interpretable scaling relations that connect a pruned LLM's post-pruning performance to its unpruned performance and pruning ratio. Across ten LLMs (1.3B-30B parameters), a 20B mixture-of-experts model, three pruning strategies (unstructured, width, and depth), and eight diverse tasks, we show that pruning laws achieve strong predictive accuracy (average extrapolation error less than 7%), reliably quantify performance degradation, and identify critical pruning thresholds beyond which recovery is infeasible. Moreover, we demonstrate that the functional form transfers across dense and mixture-of-experts architectures, pruning methods, and unseen models in zero-shot and one-shot setups, with task- and method-specific coefficients that vary in interpretable ways. These results provide both researchers and practitioners with a principled framework to select pruning strategies, estimate safe pruning ratios without exhaustive tuning, and deploy LLMs efficiently under real-world compute and latency constraints.

模型剪枝大模型压缩性能预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。