Muown实际在隐式实现角度步长衰减，新方法AngularMuown显式优化方向并可调度步长。
Muown Implicitly Performs Angular Step-size Decay
- 将权重分解为模长与方向，方向更新等价于黎曼流形上的步进
- 模长仅调节角度步长大小，解释了训练稳定性
- 新方法可调度角度步长，适合大规模Transformer模型训练
矩阵感知优化器如Muon和Muown在预训练Transformer中表现出色。特别地，Muown将每个权重矩阵拆分为行模长和未归一化的方向变量，分别用Adam和Muon更新。我们证明，Muown的方向更新等价于在归一化方向上的黎曼步进，而未归一化参数的模长仅调节角度步长。这解释了Muown的步长稳定性，并提示应显式设计角度步长。由此提出的AngularMuown直接在归一化方向上优化，并使用与径向模长更新解耦的可调度角度乘子。AngularMuown优于Muown，当前初步版本在modded nanoGPT速度竞赛的每优化器类别中领先。对Qwen2-0.5B及1.1B参数混合专家模型的实验进一步验证该算法可扩展至大模型。代码已开源：https://github.com/fhueb/angular-muown。
原文摘要 · Abstract (English)
Matrix-aware optimizers such as Muon and Muown have recently shown strong empirical performance for pre-training Transformers. In particular, Muown separates each weight matrix into row magnitudes and an un-normalized direction variable, updating the former with Adam and the latter with Muon. We show that the directional update of Muown is equivalent to a Riemannian step on the normalized directions, while the magnitude of the un-normalized parameterization only modulates the angular step size. This explains the step-size stability of Muown and suggests making the angular step size explicit. The resulting method, AngularMuown, optimizes directly over the normalized directions and uses a schedulable angular multiplier decoupled from the radial magnitude update. AngularMuown improves over Muown and, at the time of writing, a preliminary version is leading the per-optimizer category of the modded nanoGPT speedrunning competition. Further experiments on Qwen2-0.5B, and 1.1B parameter mixture-of-experts models confirm the algorithm scales beyond small models. An implementation of the algorithm is available at https://github.com/fhueb/angular-muown
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。