提出可调噪声的优化算法，收敛速度优于传统方法。
Convergence Analysis of alpha-SVRG under Strong Convexity
- 通过调节参数控制梯度噪声水平，实现更优优化路径。
- 在强凸条件下，理论证明其收敛速度超越SGD和SVRG。
- 适合追求高效训练的深度学习研究者使用。
经验风险最小化中的随机一阶方法使用采样数据近似梯度，引入噪声。为缓解此问题，近年来提出的alpha-SVRG方法可精细调控学习过程中的残余噪声水平，在现代深度学习场景中已显示优于SGD和SVRG的实验性能。本文聚焦强凸环境，推导了固定学习率下alpha-SVRG的统一收敛速率表达式，当alpha=0时退化为SGD，alpha=1时退化为SVRG。理论表明，在合适alpha取值下，alpha-SVRG收敛速度更快。线性回归上的模拟结果验证了该理论分析。
原文摘要 · Abstract (English)
Stochastic first-order methods for empirical risk minimization employ gradient approximations based on sampled data in lieu of exact gradients. Such constructions introduce noise into the learning dynamics, which can be corrected through variance-reduction techniques. There is increasing evidence in the literature that in many modern learning applications noise can have a beneficial effect on optimization and generalization. To this end, the recently proposed variance-reduction technique, alpha-SVRG [Yin et al., 2023] allows for fine-grained control of the level of residual noise in the learning dynamics, and has been reported to empirically outperform both SGD and SVRG in modern deep learning scenarios. By focusing on strongly convex environments, we first provide a unified convergence rate expression for alpha-SVRG under fixed learning rate, which reduces to that of either SGD or SVRG by setting alpha=0 or alpha=1, respectively. We show that alpha-SVRG has faster convergence rate compared to SGD and SVRG under suitable choice of alpha. Simulation results on linear regression validate our theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。