通过剪枝冗余注意力层，提升深度模型训练效率与性能。
Enhancing Layer Attention Efficiency through Pruning Redundant Retrievals
- 用KL散度量化相邻层注意力的冗余度
- 提出EBQM方法跳过冗余层，训练提速30%
- 适合追求高效训练的视觉任务研究者
越来越多证据表明，层间注意力机制显著提升了深度神经网络的架构表现。然而，现有方法存在冗余问题：相邻层学习到的注意力权重高度相似，导致多层提取近似特征，降低模型表征能力并增加训练时间。为此，本文提出一种基于相邻层间Kullback-Leibler（KL）散度的冗余量化方法，并引入增强型贝塔分位数映射（EBQM）技术，精准识别并跳过冗余层，保持模型稳定性。所提出的高效层注意力（ELA）架构在图像分类和目标检测任务中均实现性能提升，同时将训练时间减少30%。
原文摘要 · Abstract (English)
Growing evidence suggests that layer attention mechanisms, which enhance interaction among layers in deep neural networks, have significantly advanced network architectures. However, existing layer attention methods suffer from redundancy, as attention weights learned by adjacent layers often become highly similar. This redundancy causes multiple layers to extract nearly identical features, reducing the model's representational capacity and increasing training time. To address this issue, we propose a novel approach to quantify redundancy by leveraging the Kullback-Leibler (KL) divergence between adjacent layers. Additionally, we introduce an Enhanced Beta Quantile Mapping (EBQM) method that accurately identifies and skips redundant layers, thereby maintaining model stability. Our proposed Efficient Layer Attention (ELA) architecture, improves both training efficiency and overall performance, achieving a 30% reduction in training time while enhancing performance in tasks such as image classification and object detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。