通过激活余弦相似与方差设计高效低误差大模型剪枝方法
ACE: Exploring Activation Cosine Similarity and Variance for Accurate and Calibration-Efficient LLM Pruning
- 用激活向量夹角差异衡量剪枝影响，提升精度
- 结合激活方差保留语义区分，支持短序列高效剪枝
- 兼顾准确率与速度，适合资源受限场景部署
随着大语言模型（LLMs）的快速发展，其对内存和计算资源的需求急剧上升。现有剪枝方法常面临剪枝性能不佳或剪枝过程效率低的问题。本文提出一种高效且有效的剪枝方法，在保持高剪枝性能的同时显著提升剪枝速度并改善校准效率。核心创新包括：(1) 基于输出激活余弦相似度的剪枝度量，考虑密集模型与剪枝模型间输出激活的角偏差；(2) 基于激活方差的剪枝度量，有助于在剪枝后保留输出激活的语义区分能力，实现短输入序列下的有效剪枝。二者可无缝结合，全面提升剪枝的准确性和效率。实验表明，该方法在 LLaMA、LLaMA-2 与 OPT 等主流模型上，可实现最高 18% 的困惑度降低和最高 63% 的剪枝时间减少。
原文摘要 · Abstract (English)
With the rapid expansion of large language models (LLMs), the demand for memory and computational resources has grown significantly. Recent advances in LLM pruning aim to reduce the size and computational cost of these models. However, existing methods often suffer from either suboptimal pruning performance or low time efficiency during the pruning process. In this work, we propose an efficient and effective pruning method that simultaneously achieves high pruning performance and fast pruning speed with improved calibration efficiency. Our approach introduces two key innovations: (1) An activation cosine similarity loss-guided pruning metric, which considers the angular deviation of the output activation between the dense and pruned models. (2) An activation variance-guided pruning metric, which helps preserve semantic distinctions in output activations after pruning, enabling effective pruning with shorter input sequences. These two components can be readily combined to enhance LLM pruning in both accuracy and efficiency. Experimental results show that our method achieves up to an 18% reduction in perplexity and up to 63% decrease in pruning time on prevalent LLMs such as LLaMA, LLaMA-2, and OPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。