提出ReWA方法，解决稀疏优化中梯度不稳问题。
Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
- 用重参数+权重衰减+自适应学习率重构优化路径
- 在ResNet上实现比ℓ₁正则更强的稀疏性且不损失精度
- 适合追求模型轻量化又怕性能下降的研究者
稀疏优化在诸多实际应用中具有基础性挑战。常用ℓ_p正则化方法在0 < p < 1时因梯度无界易引发优化不稳定性。本文提出一种新方法ReWA，基于重参数化、权重衰减和自适应学习率，与ℓ_p正则密切相关，但揭示了不同的优化景观，有效缓解不稳定性。在CIFAR-10和ImageNet数据集上使用ResNets的实验表明，ReWA在保持测试精度的同时，显著优于ℓ₁正则化方法的稀疏性表现。
原文摘要 · Abstract (English)
Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is $\ell_p$ regularization. However, it may encounter optimization instability due to the unbounded gradients when $0<p<1$. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to $\ell_p$-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the $\ell_1$-regularization approach while preserving test accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。