提出新剪枝指标，让Transformer模型更小更稳更准
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
- 融合重要性与注意力熵，综合评估每头贡献
- 剪枝后模型质量提升15.2%，稳定性提高2.04倍
- 适合需要高效部署的NLP模型压缩场景
基于Transformer的模型在自然语言处理任务中表现卓越，但其多层结构与注意力头带来了推理与部署效率挑战。为应对这一问题，近年提出了多种剪枝方法。其中，基于梯度的注意力头重要性评分（HIS）因其可解释性、高效性及识别冗余头的能力而受到关注。然而，仅依赖HIS会忽略注意力模式的多样性。为此，本文提出新型剪枝准则HIES（Head Importance-Entropy Score），将头重要性评分与注意力熵结合，从互补角度评估每个头的贡献。实验表明，基于HIES的剪枝方法在模型质量上相比仅使用HIS的方法提升最高达15.2%，稳定性提升2.04倍，可在不牺牲准确率或稳定性的情况下实现显著模型压缩。代码将在发表后公开。
原文摘要 · Abstract (English)
Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, HIES (Head Importance-Entropy Score), which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。