arXiv:2602.08232math.OCcs.LG2026-02被引 6

提出两种高效矩阵在线学习算法,可降低计算成本并保证最优误差边界。

Adaptive Matrix Online Learning through Smoothing with Guarantees for Nonsmooth Nonconvex Optimization

  • 通过平滑核范数构造自适应势函数,设计新算法框架。
  • 两种方法均实现与Shampoo相当的误差界,计算开销更低。
  • 适用于非光滑非凸优化,为模型训练提供理论保障。

我们研究了在算子范数约束下进行矩阵变量的在线线性优化,该场景因几何结构复杂,设计数据自适应且高效的自适应算法极具挑战。目前最优的自适应遗憾界由类似Shampoo的方法达成,但其需解代价高昂的二次投影子问题。为此,我们将基于梯度的预测机制扩展至自适应矩阵在线学习,并将算法设计转化为构造核范数的平滑势函数族。我们定义了此类平滑的可接受性概念,并证明任意可接受平滑均能获得与单边Shampoo一致的遗憾界。我们实例化该框架,提出两种避免二次投影的高效方法:一种是使用高斯随机平滑的自适应跟随扰动领袖(FTPL);另一种是基于增广矩阵空间中确定性双曲平滑的跟随增强矩阵领袖(FAML)。通过分析其平滑的可接受性,我们证明两者均具闭式更新,并在常数因子内匹配单边Shampoo的遗憾界,同时显著降低计算成本。最后,借助在线转非凸转换,我们推导出两种基于矩阵的优化器:从FTPL得来的Pion,以及从FAML得来的Leon。我们证明它们在非光滑非凸设置下具有收敛保证,而流行优化器Muon缺乏此性质。

原文摘要 · Abstract (English)

We study online linear optimization with matrix variables constrained by the operator norm, a setting where the geometry renders designing data-dependent and efficient adaptive algorithms challenging. The best-known adaptive regret bounds are achieved by Shampoo-like methods, but they require solving a costly quadratic projection subproblem. To address this, we extend the gradient-based prediction scheme to adaptive matrix online learning and cast algorithm design as constructing a family of smoothed potentials for the nuclear norm. We define a notion of admissibility for such smoothings and prove any admissible smoothing yields a regret bound matching the best-known guarantees of one-sided Shampoo. We instantiate this framework with two efficient methods that avoid quadratic projections. The first is an adaptive Follow-the-Perturbed-Leader (FTPL) method using Gaussian stochastic smoothing. The second is Follow-the-Augmented-Matrix-Leader (FAML), which uses a deterministic hyperbolic smoothing in an augmented matrix space. By analyzing the admissibility of these smoothings, we show both methods admit closed-form updates and match one-sided Shampoo's regret up to a constant factor, while significantly reducing computational cost. Lastly, using the online-to-nonconvex conversion, we derive two matrix-based optimizers, Pion (from FTPL) and Leon (from FAML). We prove convergence guarantees for these methods in nonsmooth nonconvex settings, a guarantee that the popular Muon optimizer lacks.

在线学习矩阵优化非凸优化自适应算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。