基于权重谱密度理论，实现更精准的分层剪枝，大幅压缩大模型且保持性能。
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
- 利用权重矩阵谱密度形状，动态分配各层剪枝比例。
- 在LLaMA-7B上实现80%剪枝率，困惑度仍合理。
- 理论驱动剪枝，适合追求高效压缩的大模型研究者。
近期研究表明，大规模语言模型（LLMs）可移除大量参数而不影响性能，剪枝成为降低模型规模的有前景策略。现有剪枝方法通常对各层采用统一剪枝率，限制了整体剪枝能力；而现有的分层剪枝多依赖启发式规则，易导致性能下降。本文基于重尾自正则化（HT-SR）理论，特别是权重矩阵的经验谱密度（ESDs）形状，设计更优的分层剪枝策略。分析发现，不同层的可剪枝性差异显著。据此提出AlphaPruning，通过形状指标为各层分配更具理论依据的稀疏率。该方法可与多种现有剪枝方法结合使用。实验表明，AlphaPruning使LLaMA-7B达到80%稀疏率的同时保持合理困惑度，为文献中首次实现。代码已开源：https://github.com/haiquanlu/AlphaPruning。
原文摘要 · Abstract (English)
Recent work on pruning large language models (LLMs) has shown that one can eliminate a large number of parameters without compromising performance, making pruning a promising strategy to reduce LLM model size. Existing LLM pruning strategies typically assign uniform pruning ratios across layers, limiting overall pruning ability; and recent work on layerwise pruning of LLMs is often based on heuristics that can easily lead to suboptimal performance. In this paper, we leverage Heavy-Tailed Self-Regularization (HT-SR) Theory, in particular the shape of empirical spectral densities (ESDs) of weight matrices, to design improved layerwise pruning ratios for LLMs. Our analysis reveals a wide variability in how well-trained, and thus relatedly how prunable, different layers of an LLM are. Based on this, we propose AlphaPruning, which uses shape metrics to allocate layerwise sparsity ratios in a more theoretically principled manner. AlphaPruning can be used in conjunction with multiple existing LLM pruning methods. Our empirical results show that AlphaPruning prunes LLaMA-7B to 80% sparsity while maintaining reasonable perplexity, marking a first in the literature on LLMs. We have open-sourced our code at https://github.com/haiquanlu/AlphaPruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。