提出两种高效求解带约束优化问题的新算法，适用于复杂模型与深度学习。
Recursive Bound-Constrained AdaGrad with Applications to Multilevel and Domain Decomposition Minimization
- 基于自适应梯度改进，支持边界约束和不精确梯度。
- 在高概率下仅需 $O(ε^{-2})$ 次迭代即可达到 $ε$-近似最优解。
- 适用于偏微分方程与神经网络训练，计算效率显著。
本文提出两种无需目标函数信息的噪声鲁棒算法，可处理边界约束、不精确梯度，并在有二阶信息时加以利用。第一种为多层方法，利用问题的层次结构；第二种为域分解方法，涵盖标准加性Schwarz分解。两者均为无约束优化中一阶AdaGrad算法的推广。由于共享统一理论框架，给出单一收敛性/复杂度理论，证明二者在高概率下均只需 $O(ε^{-2})$ 次迭代与噪声梯度评估，即可求得带边界约束问题的 $ε$-近似一阶临界点。大量数值实验涵盖基于偏微分方程的问题与深度神经网络训练，验证了其卓越的计算效率。
原文摘要 · Abstract (English)
Two OFFO (Objective-Function Free Optimization) noise tolerant algorithms are presented that handle bound constraints, inexact gradients and use second-order information when available.The first is a multi-level method exploiting a hierarchical description of the problem and the second is a domain-decomposition method covering the standard addditive Schwarz decompositions. Both are generalizations of the first-order AdaGrad algorithm for unconstrained optimization. Because these algorithms share a common theoretical framework, a single convergence/complexity theory is provided which covers them both. Its main result is that, with high probability, both methods need at most $O(ε^{-2})$ iterations and noisy gradient evaluations to compute an $ε$-approximate first-order critical point of the bound-constrained problem. Extensive numerical experiments are discussed on applications ranging from PDE-based problems to deep neural network training, illustrating their remarkable computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。