按层动态调整稀疏率，让大模型压缩更高效。
PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning
- 根据激活值分位数动态分配每层稀疏度，不固定比例。
- 在LLaMA-2-7B上50%稀疏时，困惑度降至10.96（原为12.92）。
- 无需微调，计算开销极小，适合追求高效压缩的研究者。
一次性剪枝方法如Wanda和SparseGPT对Transformer的每一层使用相同的稀疏率，忽略了层间重要性差异。本文提出PALS（百分位感知的逐层稀疏），依据激活值的99分位数调整每层稀疏度，且稀疏率在目标值上下±5%范围内浮动。在LLaMA-2-7B上以50%稀疏率进行剪枝时，PALS达到10.96的WikiText-2困惑度，优于均匀剪枝的Wanda（12.92，9次实验均值，p < 0.001）。该优势具有架构依赖性：LLaMA-3-8B仅有小幅提升，Mistral-7B则无明显改善。我们还发现基于梯度的稀疏分配——看似更合理的方法——效果反而劣于随机，表明梯度大小无法准确预测权重移除的影响。PALS对剪枝流程开销极小，且无需微调。
原文摘要 · Abstract (English)
One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to $\pm 5\%$ around the target ratio. On LLaMA-2-7B at 50\% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, $p < 0.001$). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。