通过多层镜像下降实现动态稀疏训练，显著降低计算量且保持精度。
Sparse Training of Neural Networks based on Multilevel Mirror Descent
- 结合线性Bregman迭代与结构自适应冻结，动态调整稀疏模式。
- 相比SGD，FLOPs减少至6%，测试准确率保持不变。
- 适合追求高效训练的模型压缩与边缘部署场景。
我们提出一种基于线性化Bregman迭代/镜像下降的动态稀疏训练算法，通过交替进行静态与动态稀疏模式更新，利用自然产生的稀疏性。核心思想是将稀疏诱导的Bregman迭代与网络结构自适应冻结相结合，以高效探索稀疏参数空间并维持稀疏性。通过嵌入多层级优化框架，我们提供了收敛保证。实验证明，该方法在标准基准上可生成高度稀疏且高精度的模型。相较于标准Bregman迭代38%的FLOPs节省，本方法可将FLOPs减少至6%的同时保持测试准确率。此外，在使用稀疏感知的CPU实现时，训练时间可降低约50%。
原文摘要 · Abstract (English)
We introduce a dynamic sparse training algorithm based on linearized Bregman iterations / mirror descent that exploits the naturally incurred sparsity by alternating between periods of static and dynamic sparsity pattern updates. The key idea is to combine sparsity-inducing Bregman iterations with adaptive freezing of the network structure to enable efficient exploration of the sparse parameter space while maintaining sparsity. We provide convergence guaranties by embedding our method in a multilevel optimization framework. Furthermore, we empirically show that our algorithm can produce highly sparse and accurate models on standard benchmarks. We also show that the theoretical number of FLOPs compared to SGD training can be reduced from 38% for standard Bregman iterations to 6% for our method while maintaining test accuracy.We additionally show a training time reduction by about 50%, when using a sparsity-aware CPU implementation of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。