改进矩阵优化器，让大模型预训练更快更稳定。
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
- 通过固定权重矩阵及其更新的范数，提升优化器效率。
- 在12亿参数模型上提速20%~30%，优于标准权重衰减方法。
- 适合大规模语言模型训练，尤其关注训练稳定性与学习率迁移。
基于矩阵的优化器（如Muon）可显著加速语言模型预训练，但在模型规模和数据量增大时，其相对于AdamW的优势会因使用标准常数解耦权重衰减而减弱。本文提出Hyperball，一种简单的优化器包装器，将权重矩阵及其对应优化器更新的Frobenius范数设为固定常数，以解决此问题。在类似Qwen3的1.2B参数模型上，Muon Hyperball相比权重衰减基线实现20%~30%的令牌等效加速。此外，Hyperball在不同宽度和深度间提升了学习率迁移性能。该方法基于先前理论：权重衰减会导致一个仅依赖训练超参数的均衡权重范数，从而决定角度学习率，即权重矩阵方向变化速度。
原文摘要 · Abstract (English)
Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。