提出SparseOpt优化器，解决稀疏训练中归一化导致的梯度偏斜问题。
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
- 设计针对稀疏性的优化器，缓解批归一化对稀疏层的负面影响。
- 在ResNet上实现更快收敛，图像分类准确率接近密集训练。
- 首次系统分析归一化与稀疏训练的交互机制，适合高效模型研发者。
动态稀疏训练（DST）通过维持网络稀疏性并动态调整拓扑结构来训练神经网络。尽管计算量显著降低，但现有DST方法收敛速度远慢于密集训练，通常需要相近的训练时间才能达到相似精度。我们通过理论分析和实证研究证明，批归一化（BN）会严重阻碍稀疏训练效果。为此，本文提出SparseOpt——一种专为稀疏性设计的优化器，有效缓解该问题。在CIFAR-100和ImageNet上的实验表明，所提方法在多个ResNet模型上均实现了更快的收敛速度和更优的泛化性能。本工作揭示了当前归一化层在稀疏训练中的局限性，并首次系统研究了批归一化、稀疏层与动态稀疏训练之间的相互作用，为使稀疏训练真正媲美密集训练迈出了关键一步。
原文摘要 · Abstract (English)
Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。