让可变更新的优化算法自动求导，还能保证收敛性。
Automatic Differentiation of Optimization Algorithms with Time-Varying Updates
- 用反向传播思想处理随时间变化的优化步骤。
- 可变步长梯度下降和FISTA的导数迭代保持原算法收敛速度。
- 适合研究优化算法原理或做端到端训练的人看。
许多优化算法采用随时间变化的更新规则，例如可变的学习率、动量参数或海森矩阵近似。本文将展开（unrolled）或自动微分应用于这类时变迭代过程，并为所得导数迭代提供收敛性（速率）保证。我们调整这些收敛结果并应用于带可变步长的近端梯度下降和解决部分光滑问题的FISTA。通过求解ℓ₁与ℓ₂正则化的线性与逻辑回归问题进行数值验证。理论与实验结果表明，算法的收敛速率会反映在其导数迭代中。
原文摘要 · Abstract (English)
Numerous Optimization Algorithms have a time-varying update rule thanks to, for instance, a changing step size, momentum parameter or, Hessian approximation. In this paper, we apply unrolled or automatic differentiation to a time-varying iterative process and provide convergence (rate) guarantees for the resulting derivative iterates. We adapt these convergence results and apply them to proximal gradient descent with variable step size and FISTA when solving partly smooth problems. We confirm our findings numerically by solving $\ell_1$ and $\ell_2$-regularized linear and logisitc regression respectively. Our theoretical and numerical results show that the convergence rate of the algorithm is reflected in its derivative iterates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。