通过图论方法自动选择互补注意力头,实现高效模型压缩。
Complementary Attention Head Pruning for Efficient Transformers

- 将注意力头选择建模为全局图聚类问题,基于信息距离找互补头。
- 在不同规模模型上压缩率更高时仍保持更优性能,峰值提升3.2%。
- 避免梯度法的近输出偏好,保留中间层关键结构,适合部署优化。
基于Transformer的模型在自然语言处理中取得巨大成功,但其架构扩展导致参数量庞大,限制了在资源受限环境中的部署。尽管结构化剪枝提供了压缩路径,现有最优方法常依赖基于梯度的重要性排序或随机门控,存在不稳定性、结构退化及需大量手动调参的问题。本文提出CAHP(互补注意力头剪枝)框架,将头选择重新定义为全局图论问题。不同于孤立评估头,CAHP结合图聚类与信息论距离度量,识别并保留拓扑多样性高的互补注意力头。无需预设稀疏度或剪枝比例,框架通过识别边际性能递减曲线自动确定各层选中的头数,依据多项式阶数判定。在SST-5和MNLI基准上对多种规模Transformer模型的广泛评估表明,CAHP持续优于对比基线,尤其在高压缩率下表现突出。结构分析显示,CAHP避免了梯度法的‘邻近偏差’,即不偏向输出层附近头,而是保留功能关键的中间层头。
原文摘要 · Abstract (English)
The remarkable success of Transformer-based models in natural language processing stems from architectural scaling, which leads to a large number of parameters and hinders deployment in resource-constrained environments. While structured pruning offers a pathway to compression, existing state-of-the-art methods often rely on gradient-based importance ranking or stochastic gating, which suffer from instability, structural degeneration, and the need for extensive manual hyperparameter tuning. In this paper, we introduce CAHP (Complementary Attention Head Pruning), a novel post-hoc framework that redefines head selection as a global graph-theoretical problem. Rather than evaluating heads in isolation, CAHP utilizes graph-based clustering combined with information-theoretic distance measures to identify and preserve a topologically diverse subset of complementary attention heads. Without requiring a predefined sparsity level or pruning ratio, the framework automatically determines the number of selected attention heads across layers by identifying a diminishing marginal performance curve, where pruning additional heads leads to a sharp degradation in performance, as determined by the chosen polynomial degree. Extensive evaluations on the SST-5 and MNLI benchmarks, across different Transformer model scales, demonstrate that CAHP consistently outperforms competitive baselines, particularly in high-compression regimes. Furthermore, our structural analysis shows that CAHP avoids the "proximity bias" of gradient-based pruning methods, which tend to preserve heads mainly in layers close to the output, and instead retains a functionally critical set of attention heads in the model's intermediate layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。