提出无需调学习率的随机信任域方法,可稳定优化深层模型和约束问题。
Stochastic Trust-Region Methods for Over-parameterized Models

- 构建统一的随机信任域框架,自动调节步长,避免手动调参。
- 在强增长条件下,达到 $O(\varepsilon^{-2} \log(1/\varepsilon))$ 的迭代复杂度。
- 适用于深层神经网络训练与等式约束优化,性能媲美调优基线。
在插值型假设(如强增长条件)下,随机优化方法可达到与全批量方法相当的收敛速度,但其性能(尤其是SGD)对学习率选择仍高度敏感。为此,本文提出统一的随机信任域框架,消除手动步长调参需求,并自然扩展至等式约束问题。对于无约束优化,设计了一种一阶随机信任域算法,在强增长条件下,达到找到 $\varepsilon$-驻点所需的迭代与随机一阶预言机复杂度为 $O(\varepsilon^{-2} \log(1/\varepsilon))$。对于等式约束问题,引入基于二次惩罚的随机信任域方法,惩罚参数为 $μ$,建立达到惩罚问题 $\varepsilon$-驻点的迭代与预言机复杂度为 $O(\varepsilon^{-4} \log(1/\varepsilon))$,对应原问题的 $O(\varepsilon)$-近似KKT点。在深度神经网络训练与正交约束子空间拟合上的数值实验表明,所提方法性能可媲美精心调参的随机基线,同时表现出稳定的优化行为,且无需手动学习率调度即可有效处理硬约束。
原文摘要 · Abstract (English)
Under interpolation-type assumptions such as the strong growth condition, stochastic optimization methods can attain convergence rates comparable to full-batch methods, but their performance, particularly for SGD, remains highly sensitive to step-size selection. To address this issue, we propose a unified stochastic trust-region framework that eliminates manual step-size tuning and extends naturally to equality-constrained problems. For unconstrained optimization, we develop a first-order stochastic trust-region algorithm and show that, under the strong growth condition, it achieves an iteration and stochastic first-order oracle complexity of $O(\varepsilon^{-2} \log(1/\varepsilon))$ for finding an $\varepsilon$-stationary point. For equality-constrained problems, we introduce a quadratic-penalty-based stochastic trust-region method with penalty parameter $μ$, and establish an iteration and oracle complexity of $O(\varepsilon^{-4} \log(1/\varepsilon))$ to reach an $\varepsilon$-stationary point of the penalized problem, corresponding to an $O(\varepsilon)$-approximate KKT point of the original constrained problem. Numerical experiments on deep neural network training and orthogonally constrained subspace fitting demonstrate that the proposed methods achieve performance comparable to well-tuned stochastic baselines, while exhibiting stable optimization behavior and effectively handling hard constraints without manual learning-rate scheduling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。