提出指数力矩剪枝,高效压缩模型且精度几乎不变。
Towards Universal & Efficient Model Compression via Exponential Torque Pruning
- 用指数级力场替代线性力,精准剪除远距离冗余模块。
- 在多个领域实验中压缩率显著超越现有方法,精度损失极小。
- 适合追求高效率模型压缩的工程师与研究者使用。
现代深度神经网络规模迅速膨胀,带来计算成本和内存占用的挑战,推动高效模型压缩技术的发展。现有最先进方法采用类扭矩正则化,迫使神经模块围绕选定的枢轴点分布,但其剪枝效果不理想,剪枝后网络仍较密集,且精度下降明显。本文观察到这一问题源于默认的线性力施加方式,对不同距离的神经模块施加了不恰当的力。为高效剪除冗余且远离枢纽的模块,同时保留靠近枢纽、对推理至关重要的模块,本文提出指数力矩剪枝(Exponential Torque Pruning, ETP),采用指数级力应用方案进行正则化。大量跨领域的实验表明,尽管方法极为简单,ETP 在保持极低精度损失的同时,实现了显著高于现有最优剪枝策略的压缩率。
原文摘要 · Abstract (English)
The rapid growth in complexity and size of modern deep neural networks (DNNs) has increased challenges related to computational costs and memory usage, spurring a growing interest in efficient model compression techniques. Previous state-of-the-art approach proposes using a Torque-inspired regularization which forces the weights of neural modules around a selected pivot point. Whereas, we observe that the pruning effect of this approach is far from perfect, as the post-trained network is still dense and also suffers from high accuracy drop. In this work, we attribute such ineffectiveness to the default linear force application scheme, which imposes inappropriate force on neural module of different distances. To efficiently prune the redundant and distant modules while retaining those that are close and necessary for effective inference, in this work, we propose Exponential Torque Pruning (ETP), which adopts an exponential force application scheme for regularization. Experimental results on a broad range of domains demonstrate that, though being extremely simple, ETP manages to achieve significantly higher compression rate than the previous state-of-the-art pruning strategies with negligible accuracy drop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。