提出剪枝后微调的性能预测定律，指导高效资源分配。
P$^2$ Law: Scaling Law for Post-Training After Model Pruning
- 基于模型规模、微调数据量等四因素建立预测公式
- 在多种剪枝率和模型上验证，可准确预估损失下降
- 适合需要优化微调成本的研究者与工程人员
剪枝是降低大语言模型硬件需求的常用方法。为恢复剪枝后的模型性能，通常采用后训练来缓解性能下降。尽管后训练受益于更大数据集，但当数据量已较大时，继续增加数据带来的性能提升有限。为平衡后训练成本与模型性能，需探索最优后训练数据量。通过对 Llama-3 与 Qwen-2.5 系列模型在多种常见剪枝方法下的大量实验，我们揭示了剪枝后训练的缩放规律，称为 P² 法则。该法则识别出四个关键预测因子：剪枝前模型规模、后训练词元数量、剪枝率以及剪枝前模型损失。此外，P² 法则可推广至更大数据集、更大模型规模和更高剪枝率，为剪枝后大模型的后训练提供了重要指导。
原文摘要 · Abstract (English)
Pruning has become a widely adopted technique for reducing the hardware requirements of large language models (LLMs). To recover model performance after pruning, post-training is commonly employed to mitigate the resulting performance degradation. While post-training benefits from larger datasets, once the dataset size is already substantial, increasing the training data provides only limited performance gains. To balance post-training cost and model performance, it is necessary to explore the optimal amount of post-training data.Through extensive experiments on the Llama-3 and Qwen-2.5 series models, pruned using various common pruning methods, we uncover the scaling \textbf{Law} for \textbf{P}ost-training after model \textbf{P}runing, referred to as the P$^2$ Law.This law identifies four key factors for predicting the pruned model's post-training loss: the model size before pruning, the number of post-training tokens, the pruning rate, and the model's loss before pruning. Moreover, P$^2$ Law can generalize to larger dataset sizes, larger model sizes, and higher pruning rates, offering valuable insights for the post-training of pruned LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。