通过耦合检测动态调整学习率，提升SGD收敛稳定性。
Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent
- 用两个耦合的SGD路径距离判断是否达到最优状态。
- 实验证明在凸与非凸问题上均优于现有方法。
- 对超参数不敏感，适合实际应用中的自动化调参。
随机梯度下降（SGD）的收敛性能高度依赖学习率设置。使用固定学习率时，SGD迭代过程形成马尔可夫链，在初始阶段快速收敛；但到达稳态后，参数会在最优解附近持续振荡，不再进步。本文研究固定学习率下SGD的收敛诊断，旨在设计有效的动态学习率方案。提出一种基于耦合的收敛诊断方法，通过监测两个耦合的SGD迭代序列之间的距离来识别稳态。该诊断统计量简单且理论上能准确追踪从瞬态到稳态的转变。通过大量数值实验，对比多种现有方法，所提耦合式学习率方案在多种凸与非凸优化问题上均表现更优，且对超参数具有强鲁棒性。
原文摘要 · Abstract (English)
The convergence behavior of Stochastic Gradient Descent (SGD) crucially depends on the stepsize configuration. When using a constant stepsize, the SGD iterates form a Markov chain, enjoying fast convergence during the initial transient phase. However, when reaching stationarity, the iterates oscillate around the optimum without making further progress. In this paper, we study the convergence diagnostics for SGD with constant stepsize, aiming to develop an effective dynamic stepsize scheme. We propose a novel coupling-based convergence diagnostic procedure, which monitors the distance of two coupled SGD iterates for stationarity detection. Our diagnostic statistic is simple and is shown to track the transition from transience stationarity theoretically. We conduct extensive numerical experiments and compare our method against various existing approaches. Our proposed coupling-based stepsize scheme is observed to achieve superior performance across a diverse set of convex and non-convex problems. Moreover, our results demonstrate the robustness of our approach to a wide range of hyperparameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。