用语法结构指导注意力剪枝，让模型更省更懂。
SAP: Syntactic Attention Pruning for Transformer-based Language Models
- 结合句法结构与注意力模式，智能筛选关键注意力头。
- 无需重训练，剪枝后性能优于现有方法。
- 适合追求高效可解释性的模型压缩研究者。
本文提出语法注意力剪枝（SAP），一种针对Transformer语言模型的有效注意力头剪枝方法。不同于仅依赖权重与激活数值分析的传统方法，SAP融合句子的句法结构与注意力模式，引导剪枝过程。借助这些语言学特征，SAP不仅达到与顶尖方法相当的性能，还提升了模型行为的可解释性。为进一步增强鲁棒性，提出候选过滤（CF）机制，根据头部对模型性能的贡献度优先排序，缓解剪枝带来的性能下降。实验表明，SAP能有效保留高密度强注意力值的关键头部，在无需重训练的设置下表现优于现有剪枝策略。这些结果表明，SAP为模型压缩研究开辟了新方向，具备在各类Transformer语言模型中灵活应用的潜力。
原文摘要 · Abstract (English)
This paper introduces Syntactic Attention Pruning (SAP), a novel method for effectively pruning attention heads in Transformer models. Unlike conventional approaches that rely solely on mathematical analysis of model weights and activations, SAP incorporates both the syntactic structure and attention patterns of sentences to guide the pruning process. By leveraging these linguistic features, SAP not only achieves performance comparable to state-of-the-art methods but also enhances the interpretability of model behavior. To further improve robustness, we propose Candidate Filtering (CF), a mechanism that prioritizes heads based on their contribution to model performance, mitigating degradation during pruning. Experimental results indicate that SAP effectively preserves critical heads of a high density of strong attention values, outperforming existing head pruning strategies in retrain-free settings. These findings position SAP as a promising foundation for a new direction in model compression research, offering high flexibility for pruning across all transformer-based language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。