arXiv:2605.04230cs.LGcs.AI2026-05被引 1

提出层间最优控制框架,让优化器更懂网络几何结构。

Layerwise LQR for Geometry-Aware Optimization of Deep Networks

论文配图:Layerwise LQR for Geometry-Aware Optimization of Deep Networks
图 1 · 摘自论文原文
  • 将梯度下降转化为分层LQR问题,捕捉跨层几何关系。
  • 在ResNet和Transformer上提升优化速度与测试精度,开销小。
  • 适合追求高效二阶优化的深度学习研究者使用。

几何感知优化器如牛顿法和自然梯度可改善深度学习的条件数,但其可扩展变体(如K-FAC、Shampoo)通常早期引入结构近似,丢弃由网络计算引发的跨层交互。本文提出分层LQR(LLQR)框架,通过全局分层最优控制目标学习结构化逆预条件器。核心是发现:一类广义散度诱导二次模型下的最速下降步等价于有限时域线性二次调节器(LQR)问题。该形式揭示了编码原始稠密几何的层间动态与代价矩阵。随后,我们推导出一种可扩展松弛,通过最小化LQR目标来学习对角、(E-)Krönecker分解或其他结构化逆预条件器,并在迭代中复用。所得优化器保持与二阶几何的严谨关联,无需构建或求逆全局曲率矩阵。在ResNets和Transformers上的实验表明,LLQR改善了优化动态,常带来更高的最终测试性能,且仅增加少量运行时间开销。它确立了LLQR作为实用的几何感知二阶方法框架,并为评估可扩展近似提供了基准。

原文摘要 · Abstract (English)

Geometry-aware optimizers such as Newton and natural gradient can improve conditioning in deep learning, but scalable variants such as K-FAC, Shampoo, and related preconditioners usually impose structural approximations early, often discarding cross-layer interactions induced by the network computation. We introduce Layerwise LQR (LLQR), a framework for learning structured inverse preconditioners under a global layerwise optimal-control objective. The starting point is an exact equivalence: the steepest-descent step under a broad class of divergence-induced quadratic models--including Newton, Gauss-Newton, Fisher/natural-gradient, and intermediate-layer metrics--can be written as a finite-horizon Linear Quadratic Regulator (LQR) problem. This formulation serves as a reference that exposes the layerwise dynamics and cost matrices encoding the original dense geometry. We then derive a scalable relaxation that learns diagonal, (E-)Kronecker-factored, or other structured inverse preconditioners by minimizing the LQR objective and reusing them across iterations. The resulting optimizer wraps standard methods while retaining a principled connection to second-order geometry, without forming or inverting the global curvature matrix. Experiments on ResNets and Transformers show that LLQR improves optimization dynamics and often translates these gains into improved final test performance, while adding only modest wall-clock overhead. It establishes LLQR as a practical framework for geometry-aware second-order methods and a reference for evaluating scalable approximations.

二阶优化预条件器深度学习最优控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。