arXiv:2608.12009math.OCcs.LG2026-08

提出无需线搜索的自适应梯度优化方法,提升稳定性与收敛性。

Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size

论文配图:Adaptive Bregman Proximal Stochastic Gradient with a Stabilized Barzilai--Borwein Step Size
图 1 · 摘自论文原文
  • 结合SAGA梯度表与稳定化Barzilai-Borwein步长,自动调节学习率。
  • 在凸问题上达O(n/K)收敛率,非凸下残差为O(1/K),且对初值不敏感。
  • 适合需要低方差、高稳定性的机器学习优化场景,如逻辑回归。

Bregman近端随机梯度(BPSG)方法将方差缩减技术应用于几何特性难以用欧氏光滑性刻画的优化目标。然而其性能仍高度依赖步长:原始随机曲率估计波动剧烈,而线搜索需反复计算近端算子。本文提出Ada-BPSG,一种无需线搜索的BPSG方法,将SAGA梯度表与稳定化Barzilai--Borwein(BB)候选步长结合。通过中位数聚合增量割线信息,使局部比值接近奇异时权重极小;显式防护机制将曲率估计转化为收敛所需的有界步长序列。该设计建立了从相对光滑性与分量方差控制到有限维赋范空间收敛的直接分析链。我们证明了凸目标下O(n/K)的遍历收敛率,相对二次增长下重启线性速率,以及非凸情形下Bregman近端残差的O(1/K)界。在逻辑回归与稀疏非负矩阵分解任务上,Ada-BPSG在获得更低目标值的同时,对初始步长的敏感性显著低于标准方差缩减基线,且避免了线搜索开销。

原文摘要 · Abstract (English)

Bregman proximal stochastic gradient (BPSG) methods bring variance-reduced composite optimization to objectives whose geometry is poorly captured by Euclidean smoothness. Their performance, however, remains sensitive to the step size: raw stochastic curvature estimates can fluctuate sharply, whereas line searches add repeated proximal evaluations. We introduce Ada-BPSG, a line-search-free BPSG method that couples the SAGA gradient table with a stabilized Barzilai--Borwein (BB) candidate. A mediant aggregates incremental secant information so that nearly singular local ratios receive little weight, and an explicit safeguard translates the resulting curvature estimate into the bounded step-size sequence required for convergence. This design yields a direct analytical chain from relative smoothness and component-wise variance control to convergence in finite-dimensional normed spaces. We prove an $O(n/K)$ ergodic rate for convex objectives, a restarted linear rate under relative quadratic growth, and an $O(1/K)$ bound for a Bregman proximal residual in the nonconvex setting. On logistic regression and sparse nonnegative matrix factorization, Ada-BPSG combines low objective values with substantially less sensitivity to the initial step size than standard variance-reduced baselines, while avoiding line search.

优化算法随机梯度自适应步长

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。