改进梯度下降的偏差问题，提升模型收敛精度。
Reducing Bias in Deep Learning Optimization: The RSGDM Approach
- 用微分修正项消除梯度平均的偏差和延迟。
- 在CIFAR数据集上收敛精度优于SGDM。
- 适合追求优化器稳定性的深度学习研究者。
目前广泛使用的一阶深度学习优化器包括非自适应学习率优化器和自适应学习率优化器。前者以SGDM（带动量的随机梯度下降）为代表，后者以Adam为代表。这两种方法均使用指数移动平均估计整体梯度，但该方法存在偏差且具有滞后性。本文提出基于微分校正的RSGDM算法。主要贡献有三：1）分析了指数移动平均在SGDM算法中引入的偏差与滞后；2）引入微分估计项以纠正SGDM中的偏差与滞后，提出RSGDM算法；3）在CIFAR数据集上的实验表明，RSGDM算法在收敛精度上优于SGDM算法。
原文摘要 · Abstract (English)
Currently, widely used first-order deep learning optimizers include non-adaptive learning rate optimizers and adaptive learning rate optimizers. The former is represented by SGDM (Stochastic Gradient Descent with Momentum), while the latter is represented by Adam. Both of these methods use exponential moving averages to estimate the overall gradient. However, estimating the overall gradient using exponential moving averages is biased and has a lag. This paper proposes an RSGDM algorithm based on differential correction. Our contributions are mainly threefold: 1) Analyze the bias and lag brought by the exponential moving average in the SGDM algorithm. 2) Use the differential estimation term to correct the bias and lag in the SGDM algorithm, proposing the RSGDM algorithm. 3) Experiments on the CIFAR datasets have proven that our RSGDM algorithm is superior to the SGDM algorithm in terms of convergence accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。