arXiv:2607.22444cs.LGcs.AI2026-07被引 2

Hyperball优化器优势源于学习率调度,而非更新方向本身。

Hyperball May Not Be a Free Lunch

  • 提出角度有效学习率,分解更新为径向与切向分量
  • 数值实验表明径向更新对角度变化影响有限
  • 优化器差异主要来自有效步长演化,需精心设计调度

针对尺度不变深度网络,Hyperball类优化器在大规模训练中表现优异,通过固定矩阵参数范数并归一化更新实现。然而其优势来源尚不明确。本文从连续参数状态间的角位移出发,推导出考虑参数更新角度、参数范数和更新范数的角有效学习率,并证明传统范数度量是参数更新正交时的特例。进一步将优化器更新分解为径向与切向分量,分析径向更新对单步角位移的影响。在所考虑的训练配置下,数值结果表明径向分量对角有效学习率仅有有限直接影响,无法解释MuonH早期收敛慢但后期反超MuonWD的现象。为此设计启发式实验,仅调整学习率调度以使各优化器动力学复现对方。结果表明两者主要差异源于有效步长演化,而非由Hyperball诱导的内在优越更新方向。预训练实验还显示,更激进的学习率衰减虽可加速MuonH初期训练,但可能损害后期性能。因此,保持恒定角速度并未消除学习率调度问题,合理调度仍是发挥Hyperball类优化器潜力的关键。代码已开源。

原文摘要 · Abstract (English)

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

优化器学习率调度深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。