提出新型缪子优化器，通过几何视角提升参数更新效率。
Muse: Representation Geometry of Muon Beyond Normalized Momentum
- 基于矩阵范数的极坐标映射，统一多种表示下的优化几何
- 短维度决定奇异通道数与收敛常数，影响模型性能
- 非原生表示可媲美原生表示，降低维度则趋近于归一化动量
缪子风格优化器对矩阵动量应用极坐标映射，但其更新还依赖于参数块在正交化前的表示形式。本文将这种表示选择视为优化器几何，并提出{ extmethod}，一类共享相同动量规则和Newton-Schulz后端的缪子型优化器，适用于原生、最近平方、细长及向量表示。每种Frobenius等距表示均诱导出不同的极坐标最速下降几何，其中较短矩阵维度决定支持的奇异通道数、拉回缩放系数以及随机非凸收敛界中的常数。在教师-学生模型中，曲率坍缩与各向同性Marchenko-Pastur谱分布连接早期耗散与所表示的核范数到平方Frobenius范数之比。在LLaMA2-130M和LLaMA2-600M上的预训练实验及固定动量诊断显示，平衡的非原生表示可达到与原生表示相当的性能；而减少短维度会削弱缩放能力和奇异通道支持，导致行为逐渐趋近于归一化动量。
原文摘要 · Abstract (English)
Muon-style optimizers apply a polar map to matrix momentum, but their updates also depend on the representation of each parameter block before orthogonalization. We study this representation choice as a form of optimizer geometry and introduce {\method}, a family of Muon-style optimizers that shares the same momentum rule and Newton--Schulz backend across native, nearest-square, skinny, and vector representations. Each Frobenius-isometric representation induces a distinct polar steepest-descent geometry, in which the shorter matrix dimension determines the number of supported singular channels, the pullback scaling, and the constants in stochastic nonconvex convergence bounds. In a teacher--student model, curvature collapse and an isotropic Marchenko--Pastur spectral profile connect early-stage dissipation to the represented nuclear-to-squared-Frobenius norm ratio. Pretraining experiments on LLaMA2-130M and LLaMA2-600M, together with fixed-momentum diagnostics, show that balanced non-native representations can match the performance of the native representation, whereas reducing the shorter dimension weakens the scaling and singular-channel support, leading to behavior that increasingly resembles normalized momentum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。