通过双泰勒展开与注意力分布感知，实现大模型高效稀疏化
$D^2Prune$: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awareness
- 基于双泰勒展开联合建模权重与激活扰动，提升误差估计精度
- 动态更新注意力分布，保持长尾特性，提升剪枝后模型性能
- 适用于多种大模型，尤其适合关注推理效率的部署场景
大型语言模型（LLMs）因计算需求巨大而面临部署挑战。虽然剪枝是有效的压缩方法，但现有方法存在两大缺陷：一是忽略校准数据与测试数据间激活分布的变化，导致误差估计不准；二是忽视注意力模块中激活的长尾分布特征。为此，本文提出一种新型剪枝方法 $D^2Prune$。首先，采用基于双泰勒展开的方法，联合建模权重与激活扰动，实现更精确的误差估计，从而指导剪枝掩码选择与权重更新，最小化剪枝过程中的误差。其次，提出注意力感知的动态更新策略，通过联合最小化注意力分布的KL散度与重构误差，保留注意力的长尾模式。大量实验表明，$D^2Prune$ 在多种LLM（如OPT-125M、LLaMA2/3、Qwen3）上均优于当前最优方法。此外，该动态注意力更新机制在视觉模型如DeiT上也表现良好，在ImageNet-1K上达到更优准确率。
原文摘要 · Abstract (English)
Large language models (LLMs) face significant deployment challenges due to their massive computational demands. % While pruning offers a promising compression solution, existing methods suffer from two critical limitations: (1) They neglect activation distribution shifts between calibration data and test data, resulting in inaccurate error estimations; (2) They overlook the long-tail distribution characteristics of activations in the attention module. To address these limitations, this paper proposes a novel pruning method, $D^2Prune$. First, we propose a dual Taylor expansion-based method that jointly models weight and activation perturbations for precise error estimation, leading to precise pruning mask selection and weight updating and facilitating error minimization during pruning. % Second, we propose an attention-aware dynamic update strategy that preserves the long-tail attention pattern by jointly minimizing the KL divergence of attention distributions and the reconstruction error. Extensive experiments show that $D^2Prune$ consistently outperforms SOTA methods across various LLMs (e.g., OPT-125M, LLaMA2/3, and Qwen3). Moreover, the dynamic attention update mechanism also generalizes well to ViT-based vision models like DeiT, achieving superior accuracy on ImageNet-1K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。