针对大模型注意力头剪枝，提出高层注意力精准剪裁与动态缩放方案。
High-Layer Attention Pruning with Rescaling
- 按网络层级分层剪枝，优先处理高层注意力头。
- 引入自适应缩放参数，补偿剪枝后表示幅度变化。
- 在生成任务上显著优于现有方法，适合部署优化场景。
剪枝是压缩大语言模型、降低推理延迟的有效方法。然而，传统无训练结构化剪枝方法常使用启发式度量，不加区分地移除各层的注意力头，未考虑其在网络中的位置。本文提出一种新型剪枝算法，针对性地在模型高层剪除注意力头。由于剪枝会改变令牌表示的幅度，我们引入自适应重缩放参数，校准剪枝后的表示尺度以抵消该影响。我们在多种LLM上进行实验，包括LLaMA3.1-8B、Mistral-7B-v0.3、Qwen2-7B和Gemma2-9B，覆盖27个数据集上的生成与判别任务。结果一致表明，本方法优于现有结构化剪枝方法，尤其在生成任务中表现突出。代码已公开于https://github.com/SongtaoLiu0823/HARP。
原文摘要 · Abstract (English)
Pruning is a highly effective approach for compressing large language models (LLMs), significantly reducing inference latency. However, conventional training-free structured pruning methods often employ a heuristic metric that indiscriminately removes some attention heads across all pruning layers, without considering their positions within the network architecture. In this work, we propose a novel pruning algorithm that strategically prunes attention heads in the model's higher layers. Since the removal of attention heads can alter the magnitude of token representations, we introduce an adaptive rescaling parameter that calibrates the representation scale post-pruning to counteract this effect. We conduct comprehensive experiments on a wide range of LLMs, including LLaMA3.1-8B, Mistral-7B-v0.3, Qwen2-7B, and Gemma2-9B. Our evaluation includes both generation and discriminative tasks across 27 datasets. The results consistently demonstrate that our method outperforms existing structured pruning methods. This improvement is particularly notable in generation tasks, where our approach significantly outperforms existing baselines. Code is available at https://github.com/SongtaoLiu0823/HARP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。