改进缪子优化器，用数据相关几何提升训练稳定性。
Second-Order Muon Done Right: A Principled Marriage of Spectral Geometry and Curvature
- 引入GO-MUON，通过匹配数据的几何结构复用优化步骤。
- 在软标签交叉熵下，后向因子逼近模型费舍尔与广义高斯-牛顿矩阵。
- 四步刷新近似保持缓慢变化几何的追踪延迟，适合计算与统计权衡场景。
缪子的极化更新在无权重谱几何下是精确的。我们提出GO-MUON，采用匹配数据的依赖性几何，并在多个优化步骤中复用该几何。在任意正定左右映射条件下，其原始更新精确求解对应的加权谱预言机；该结论与映射估计方式或刷新频率无关。针对softmax交叉熵损失，我们量化了观测标签反向因子趋近于模型费舍尔信息与广义高斯-牛顿矩阵的条件。此外，四步刷新机制几乎保持缓慢变化几何的追踪延迟，同时增加稳态噪声，表明懒惰几何是计算与统计间的权衡，而非降噪手段。
原文摘要 · Abstract (English)
Muon's polar update is exact for an unweighted spectral geometry. We introduce GO-MUON, which uses a matched data-dependent geometry and reuses it across several optimization steps. Conditioned on any positive-definite left and right maps, its raw update exactly solves the corresponding weighted spectral oracle; this statement is independent of how the maps are estimated or how recently they were refreshed. For softmax cross-entropy, we quantify when the observed-label backward factor approaches the model Fisher and generalized Gauss--Newton factor. We also show that four-step refresh nearly preserves the tracking delay of slowly changing geometry while increasing stationary factor noise, making lazy geometry a compute--statistics tradeoff rather than a denoising mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。