arXiv:2503.22451cs.LG2025-03被引 2

用输入数据标准差做剪枝,比现有方法更稳定有效

STADE: Standard Deviation as a Pruning Metric

  • 基于输入标准差设计新剪枝策略,理论推导更普适
  • 在Llama和OPT模型上验证,不同训练条件下性能更优
  • 适合追求高效剪枝且无需重训的AI研发人员

大型语言模型(LLMs)因训练时间长、参数量大,成为剪枝技术的理想对象。传统方法需剪枝后重训以维持性能,而先进方法如Wanda可无重训剪枝,效率更高。本文对Wanda的有效性提供理论解释,并发现其在特定机器学习场景下为最优。进一步分析表明,当该条件不满足时,原有方法不再最优,由此提出基于输入标准差的新剪枝方法STADE。理论分析显示,STADE具有更广适应性。在Llama和OPT模型上的大量实验验证了理论预测:根据训练条件不同,Wanda的最优性确实变化,而STADE表现更稳健。代码已开源。

原文摘要 · Abstract (English)

Recently, Large Language Models (LLMs) have become very widespread and are used to solve a wide variety of tasks. To successfully handle these tasks, LLMs require longer training times and larger model sizes. This makes LLMs ideal candidates for pruning methods that reduce computational demands while maintaining performance. Previous methods require a retraining phase after pruning to maintain the original model's performance. However, state-of-the-art pruning methods, such as Wanda, prune the model without retraining, making the pruning process faster and more efficient. Building upon Wanda's work, this study provides a theoretical explanation of why the method is effective and leverages these insights to enhance the pruning process. Specifically, a theoretical analysis of the pruning problem reveals a common scenario in Machine Learning where Wanda is the optimal pruning method. Furthermore, this analysis is extended to cases where Wanda is no longer optimal, leading to the development of a new method, STADE, based on the standard deviation of the input. From a theoretical standpoint, STADE demonstrates better generality across different scenarios. Finally, extensive experiments on Llama and Open Pre-trained Transformers (OPT) models validate these theoretical findings, showing that depending on the training conditions, Wanda's optimal performance varies as predicted by the theoretical framework. These insights contribute to a more robust understanding of pruning strategies and their practical implications. Code is available at: https://github.com/Coello-dev/STADE/

模型剪枝大模型优化理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。