提出一种自适应步长优化方法，降低内存占用且性能媲美Adam。
Stochastic Approximation with Block Coordinate Optimal Stepsizes
- 基于块坐标方向的二阶矩在线估计，动态调整步长。
- 收敛至目标点附近小邻域，邻域大小由估计偏差与方差决定。
- 无需凸性或光滑性假设，适合广泛优化场景。
我们研究带有块坐标步长的随机逼近问题，提出旨在最小化下一迭代点到未知目标点期望距离的自适应步长规则。这些规则利用每个块坐标的搜索方向二阶矩的在线估计。流行的Adam算法可视为其中一种特定估计器的变体。通过采用简单的条件估计器，我们推导出一种新方法，在性能上可与Adam竞争，但所需内存更少、超参数更少。我们证明该类方法几乎必然收敛至目标点的小邻域，邻域半径取决于二阶矩估计器的偏差与方差。分析基于一个简单的瞄准条件，不依赖凸性或光滑性，具有广泛适用性。
原文摘要 · Abstract (English)
We consider stochastic approximation with block-coordinate stepsizes and propose adaptive stepsize rules that aim to minimize the expected distance from the next iterate to an (unknown) target point. These stepsize rules employ online estimates of the second moment of the search direction along each block coordinate. The popular Adam algorithm can be interpreted as a variant with a specific estimator. By leveraging a simple conditional estimator, we derive a new method that obtains competitive performance against Adam but requires less memory and fewer hyper-parameters. We prove that this family of methods converges almost surely to a small neighborhood of the target point, and the radius of the neighborhood depends on the bias and variance of the second-moment estimator. Our analysis relies on a simple aiming condition that assumes neither convexity nor smoothness, thus has broad applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。