F-WANDA让大模型剪枝更省能，效果还比现有方法更好。
F-WANDA: Fisher-Reweighted Post-Training Pruning for Sustainable Deployment of Large Language Models
- 按经验费雪信息重分配每行保留参数，提升剪枝精度。
- 在LLAMA-2-7B上达到6.85的困惑度，5次提示下MMLU提升1.6个百分点。
- 剪枝耗时仅为SPARSEGPT的1/3，适合资源受限场景部署。
一次性后训练剪枝是压缩大语言模型最节能的方法，但现有方法要么牺牲质量（如WANDA），要么计算成本高（如SPARSEGPT）。本文提出F-WANDA，作为WANDA的即插即用改进，根据预激活的实证费雪信息，按比例重新分配每行保留参数的数量。该信息仅需在原有校准语料上进行一次额外反向传播即可获取，不更新任何权重。在LLAMA-2-7B模型上以50%非结构化稀疏度剪枝时，F-WANDA在WikiText-2上达到6.85的困惑度，保持与WANDA相当的流畅性，并在5次提示的MMLU任务中相比WANDA提升1.6个百分点，相比SPARSEGPT提升1.1个百分点；同时剪枝耗时仅为SPARSEGPT的三分之一，能耗更低。该方法无需额外校准数据或微调，实现了质量与剪枝成本的帕累托最优，推动大模型可持续部署。
原文摘要 · Abstract (English)
One-shot post-training pruning is the most energy-frugal compression strategy for largelanguage models (LLMs), yet existing approaches trade either quality (WANDA) or compute cost (SPARSEGPT). We introduce F-WANDA, a drop-in modification of WANDA that reallocates the per-row keep budget across output neurons in proportion to the empirical Fisher information of the pre-activation. The Fisher signal is collected in a single additional backward pass over the same calibration corpus WANDA already uses; no weights are updated. On LLAMA-2-7B at 50 % unstructured sparsity, F-WANDA attains WikiText-2 perplexity of 6.85, matches WANDA fluency, and improves 5-shot MMLU by +1.6 pp over WANDA and +1.1 pp over SPARSEGPT, while incurring only one-third of SPARSEGPT pruning wall-clock and energy. The headline trade-off is achieved without extra calibration data or fine-tuning, placing F-WANDA on the Pareto frontier of quality versus pruning cost for sustainable LLM compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。