动态重估注意力头重要性，提升剪枝精度与模型效率。
Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruning
- 每轮剪枝后重新计算梯度矩阵范数，实时更新头重要性。
- 在多个BERT变体上剪掉50%头仍保持高准确率，优于传统熵方法。
- 适合追求低功耗部署的模型压缩研究者与工程师。
注意力头剪枝已成为变压器模型压缩的有效手段，契合绿色AI时代需求。然而，现有方法多依赖静态重要性评分，无法捕捉剪枝过程中注意力头角色的动态变化。本文提出贪婪梯度范数（Greedy-Gnorm），一种新型头剪枝算法，在每次剪枝后动态重算头的重要性。具体而言，每个头的得分基于其Q/K/V梯度块的L2范数的逐元素乘积，该估计来自保留验证集，并在每轮贪心迭代中更新。这种动态评分机制缓解了过时排名问题，更真实反映剪枝进程中的梯度驱动重要性。在BERT、ALBERT、RoBERTa和XLM-RoBERTa上的大量实验表明，Greedy-Gnorm在大幅剪除注意力头时持续保持准确率，优于注意力熵方法。通过有效减小模型规模同时维持任务性能，Greedy-Gnorm为更节能的变压器模型部署提供了可行路径。
原文摘要 · Abstract (English)
Attention head pruning has emerged as an effective technique for transformer model compression, an increasingly important goal in the era of Green AI. However, existing pruning methods often rely on static importance scores, which fail to capture the evolving role of attention heads during iterative removal. We propose Greedy-Gradient norm (Greedy-Gnorm), a novel head pruning algorithm that dynamically recalculates head importance after each pruning step. Specifically, each head is scored by the elementwise product of the l2-norms of its Q/K/V gradient blocks, as estimated from a hold-out validation set and updated at every greedy iteration. This dynamic approach to scoring mitigates against stale rankings and better reflects gradient-informed importance as pruning progresses. Extensive experiments on BERT, ALBERT, RoBERTa, and XLM-RoBERTa demonstrate that Greedy-Gnorm consistently preserves accuracy under substantial head removal, outperforming attention entropy. By effectively reducing model size while maintaining task performance, Greedy-Gnorm offers a promising step toward more energy-efficient transformer model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。