arXiv:2603.13418cs.LGcs.AI2026-03

提出新剪枝方法,提升大模型在不同任务间的泛化能力

GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models

  • 按神经元跨数据集行为差异分组,避免统一评分导致偏差
  • 高稀疏度下仍保持良好性能,减少对重要性指标的依赖
  • 适合追求模型泛化与压缩效率的研究者和工程师

结构化剪枝广泛用于压缩大语言模型,但其性能严重依赖神经元重要性的估计。现有方法多基于单一校准集的激活统计,引入校准偏差,降低跨任务泛化能力。我们观察到神经元在不同数据分布下的敏感性存在异质性:部分神经元在多个校准集上排名稳定,部分则变化显著。忽略这种差异,现有方法在统一空间中对所有神经元评分,导致校准特定神经元主导排名,弱激活神经元评分不可靠。为此,我们提出 GPrune-LLM,一种基于神经元跨分布行为的结构化剪枝框架,通过重构神经元排名空间为行为一致的局部空间,自适应调整不可靠校准信号的评分来源,并在全局预算下学习模块级稀疏分配。多模型、多下游任务实验表明,GPrune-LLM 提升了基础剪枝指标的泛化能力,在高稀疏度下增益最明显,且降低了对重要性度量选择的依赖。

原文摘要 · Abstract (English)

Structured pruning is widely applied to compress large language models (LLMs), but its performance depends heavily on how neuron importance is estimated. Most existing methods rely on activation statistics from a single calibration set, which introduces calibration bias and degrades downstream cross-task generalization. We observe that neurons exhibit heterogeneous distribution sensitivity, ranging from maintaining relatively stable rankings across calibration datasets to showing substantially larger cross-dataset variation. Ignoring this heterogeneity, existing methods rank all neurons in shared spaces with a uniform scoring source, so calibration-specific neurons dominate the ranking and weakly-activated neurons are scored unreliably. To address this, we propose GPrune-LLM, a structured pruning framework that reduces calibration bias by measuring and exploiting the cross-distribution behavior of neurons for fair comparison. Specifically, we restructure the neuron ranking space into behavior-consistent local spaces, adapt the scoring source where the calibration signal is unreliable, and learn per-module sparsity allocation under a global budget. Experiments across multiple models and downstream tasks show that GPrune-LLM improves the generalization of its base pruning metrics, with gains most pronounced at high sparsity, and reduces dependence on the choice of importance metric.

大模型压缩结构化剪枝泛化能力神经元重要性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。