通过融合多信号策略，显著提升变压器模型压缩效果。
Strategic Fusion Optimizes Transformer Compression
- 用线性回归与随机森林融合多种剪枝信号，优化剪枝决策。
- 随机森林融合在9个数据集中有7个表现更优，3个几乎最优。
- 结合知识蒸馏后，模型精度提升且大小比优化18.84倍。
本研究系统评估了在九个不同数据集上对变压器模型进行层剪枝的14种策略，包括基于层激活、互信息、梯度、权重和注意力等不同信号的12种策略。为克服单一信号策略的局限性，提出两种融合策略：线性回归与随机森林，实现多信号协同剪枝（战略融合）。同时引入知识蒸馏以缓解剪枝带来的精度损失。结果表明，随机森林融合策略在9个数据集中有7个表现优于单信号策略，其余两个接近最优。经蒸馏后的模型在6个数据集中超越原始精度，其余3个有效抑制精度下降。跨所有数据集，知识蒸馏平均提升精度-尺寸比18.84倍。基于数学基础与生物类比，研究证实合理融合多信号可实现资源受限场景下的高效高性能模型。
原文摘要 · Abstract (English)
This study investigates transformer model compression by systematically pruning its layers. We evaluated 14 pruning strategies across nine diverse datasets, including 12 strategies based on different signals obtained from layer activations, mutual information, gradients, weights, and attention. To address the limitations of single-signal strategies, we introduced two fusion strategies, linear regression and random forest, which combine individual strategies (i.e., strategic fusion), for more informed pruning decisions. Additionally, we applied knowledge distillation to mitigate any accuracy loss during layer pruning. Our results reveal that random forest strategic fusion outperforms individual strategies in seven out of nine datasets and achieves near-optimal performance in the other two. The distilled random forest surpasses the original accuracy in six datasets and mitigates accuracy drops in the remaining three. Knowledge distillation also improves the accuracy-to-size ratio by an average factor of 18.84 across all datasets. Supported by mathematical foundations and biological analogies, our findings suggest that strategically combining multiple signals can lead to efficient, high-performing transformer models for resource-constrained applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。