提出PolarAdamW，分离矩阵优化中谱控制与基底不变性，提升Transformer训练效果。
PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation
- 将极坐标谱控与坐标预条件结合，保留谱控制但放弃基底不变性。
- 在ImageNet子集上比Muon高1.93个百分点，比AdamW高9.5个百分点。
- 适合标准Transformer训练，对多重复结构有特殊优势的场景需谨慎使用。
Muon的矩阵级更新同时包含通过极坐标映射实现的谱控制和在多重性空间基变换下的等价性（Schur规范不变性）。本文提出PolarAdamW，将两者分离：保留Muon的极坐标谱范数控制，但打破规范不变性，因为AdamW的坐标预条件算子依赖于基底。算法上，PolarAdamW在AdamW的预条件方向上应用Muon的Newton-Schulz极坐标映射，而非原始动量，每轮迭代耗时与Muon相当。理论上证明：Muon的极坐标步骤对多重性矩阵具有Schur规范不变性，而AdamW的坐标步骤则不具备。在从头训练的DeiT-Tiny模型上，使用四个独立采样的ImageNet-1k 100类子集，PolarAdamW平均测试准确率较Muon提升1.93个百分点，较AdamW提升9.5个百分点；在300轮的DeiT式训练策略下，仍分别领先1.37和5.80个百分点。而在SO(3)等变的3D点云回归任务中，多重性基底自由度非平凡，结果反转：Muon在所有评估容量下均优于PolarAdamW，且差距随容量增大而扩大。两类矩阵极坐标优化器始终优于AdamW。该双分离实验揭示：谱控制与规范不变性可解耦——前者与AdamW预条件兼容，后者仅在多重性基底自由度具有结构性意义时才重要。
原文摘要 · Abstract (English)
Muon's matrix-level update couples two distinct effects: spectral control via a polar map, and equivariance under orthogonal changes of multiplicity-space basis (Schur gauge-equivariance). We separate them with PolarAdamW, a controlled hybrid that preserves Muon's polar spectral-norm control but breaks the gauge-equivariance, since AdamW's coordinatewise preconditioner is basis-dependent. Algorithmically, PolarAdamW applies Muon's Newton-Schulz polar map to AdamW's preconditioned direction rather than to raw momentum, at per-iteration wall-time comparable to Muon. We prove that Muon's polar step is Schur gauge-equivariant on multiplicity matrices while AdamW's coordinatewise step is not. On DeiT-Tiny trained from scratch on four independently sampled 100-class subsets of ImageNet-1k, where multiplicity-basis freedom is trivial, PolarAdamW outperforms Muon by +1.93 pp in test accuracy on average and AdamW by +9.5 pp; under the 300-epoch DeiT-style recipe, it remains ahead of Muon by +1.37 pp and AdamW by +5.80 pp on average. On SO(3)-equivariant 3D point-cloud regression, where multiplicity-basis freedom is non-trivial, the ordering reverses: Muon outperforms PolarAdamW at every audited capacity, and the gap widens with capacity. Both matrix-polar optimisers continue to outperform AdamW. This double dissociation separates spectral control from Schur gauge-equivariance: the first composes well with AdamW preconditioning on standard transformers, while the second becomes consequential when multiplicity-basis freedom is structurally non-trivial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。