用区域梯度剪枝大模型，速度更快精度更高。
Wanda++: Pruning Large Language Models via Regional Gradients
- 基于解码器块级区域梯度设计剪枝评分
- 在语言建模任务中困惑度提升32%
- 支持轻量剪枝且兼容LoRA微调
大语言模型剪枝旨在移除不重要权重以提升推理速度，同时保持精度。现有方法常因缺乏全模型稀疏性感知微调导致精度下降。本文提出Wanda++，首次利用解码器块级区域梯度改进剪枝评分，并设计高效区域优化方法，降低稠密与稀疏模型输出差异。实验表明，相比Wanda，Wanda++在语言建模任务中困惑度最高提升32%，并在下游任务中表现良好。尽管使用区域优化更新权重，该方法仍与稀疏性感知微调正交，可显著降低困惑度。本方法轻量高效，在单张H100 GPU上可在10分钟内完成7B LLaMA模型的剪枝。
原文摘要 · Abstract (English)
Large Language Models (LLMs) pruning seeks to remove unimportant weights for inference speedup with minimal accuracy impact. However, existing methods often suffer from accuracy degradation without full-model sparsity-aware fine-tuning. This paper presents Wanda++, a novel pruning framework that outperforms the state-of-the-art methods by utilizing decoder-block-level \textbf{regional} gradients. Specifically, Wanda++ improves the pruning score with regional gradients for the first time and proposes an efficient regional optimization method to minimize pruning-induced output discrepancies between the dense and sparse decoder output. Notably, Wanda++ improves perplexity by up to 32\% over Wanda in the language modeling task and generalizes effectively to downstream tasks. Moreover, despite updating weights with regional optimization, Wanda++ remains orthogonal to sparsity-aware fine-tuning, further reducing perplexity with LoRA in great extend. Our approach is lightweight, pruning a 7B LLaMA model in under 10 minutes on a single H100 GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。