无需数据即可快速压缩大模型，剪掉冗余注意力层提升推理速度。
Data-Free Pruning of Self-Attention Layers in LLMs
- 基于查询-键耦合度排序，仅用权重判断注意力层重要性。
- 剪掉8到16层后推理速度提升1.3倍，零样本准确率损失小于2%。
- 无需数据、无需前向传播，适合实际部署中的轻量化需求。
大型语言模型中的许多自注意力子层可被移除而几乎不损失性能。我们提出注意力抑制假说：预训练中部分深层注意力层会抑制自身贡献，由残差流和MLP承载表征。本文提出Gate-Norm，一种仅需权重的单次评分方法，通过查询-键耦合度对注意力子层排序并移除耦合最弱者，无需校准数据、前向传播、微调或专用内核。在40层、130亿参数的LLaMA模型上，可在一秒内完成剪枝。剪除8至16个注意力子层可使推理吞吐量最高提升1.30倍，同时在BoolQ、RTE、HellaSwag、WinoGrande、ARC-Easy/Challenge和OpenBookQA等任务上，平均零样本准确率与未剪枝基线相差不超过2%。在各项设置中，Gate-Norm在准确率上媲美数据驱动剪枝方法，但评分速度提升约1000倍，实现高效、无数据的LLM压缩。
原文摘要 · Abstract (English)
Many self-attention sublayers in large language models (LLMs) can be removed with little to no loss. We attribute this to the Attention Suppression Hypothesis: during pre-training, some deep attention layers learn to mute their own contribution, leaving the residual stream and the MLP to carry the representation. We propose Gate-Norm, a one-shot, weight-only criterion that ranks attention sublayers by query--key coupling and removes the least coupled ones, requiring no calibration data, no forward passes, no fine-tuning, and no specialized kernels. On 40-layer, 13B-parameter LLaMA models, Gate-Norm prunes the model in under a second. Pruning $8$--$16$ attention sublayers yields up to $1.30\times$ higher inference throughput while keeping average zero-shot accuracy within $2\%$ of the unpruned baseline across BoolQ, RTE, HellaSwag, WinoGrande, ARC-Easy/Challenge, and OpenBookQA. Across these settings, Gate-Norm matches data-driven pruning methods in accuracy while being $\sim 1000\times$ faster to score layers, enabling practical, data-free compression of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。