打破几何迷信:随机或反向谱也能媲美顶级优化器
Muon is Not That Special: Random or Inverted Spectra Work Just as Well

- 用舒坦范数族构造新优化器Freon,可平滑过渡到拟范数区
- 随机替换奇异值的Kaon表现不输Muon,证明几何非关键
- 性能由对齐度与下降势决定,步长调优比几何结构更重要
近期Muon优化器的成功重新激发了对非欧几里得优化的兴趣,通常以二阶方法和线性最小化预言机(LMO)理论为依据。本文通过三项贡献挑战这一几何叙事,证明精确几何结构并非影响优化性能的关键因素。首先,提出基于舒坦(quasi-)范数的Freon优化器族,采用新型可证明最优的QDWH迭代逼近方法。Freon在SGD与Muon间自然插值,并平滑外推至拟范数区域。实验证明,GPT-2的最佳舒坦参数严格位于拟范数区间,无法被任何酉不变的LMO表示。其次,注意到Freon在多种指数下表现良好,引入极端优化器Kaon——将奇异值替换为随机噪声。尽管缺乏任何一致几何结构,Kaon仍达到与Muon相当的性能,并保持经典收敛保证,证明严格遵循精确几何在实践中无关紧要。第三,我们揭示性能实际上由两个局部量控制:对齐度与下降势。每个优化器必须围绕这两个量调整步长。虽然其动态难以先验预测,但在随机特征模型中评估表明,Muon成功并非因追踪理想全局几何,而是保障了步长最优性。
原文摘要 · Abstract (English)
The recent empirical success of the Muon optimizer has renewed interest in non-Euclidean optimization, typically justified by similarities with second-order methods, and linear minimization oracle (LMO) theory. In this paper, we challenge this geometric narrative through three contributions, demonstrating that precise geometric structure is not the key factor affecting optimization performance. First, we introduce Freon, a family of optimizers based on Schatten (quasi-)norms, powered by a novel, provably optimal QDWH-based iterative approximation. Freon naturally interpolates between SGD and Muon, while smoothly extrapolating into the quasi-norm regime. Empirically, the best-performing Schatten parameters for GPT-2 lie strictly within the quasi-norm regime, and thus cannot be represented by any unitarily invariant LMO. Second, noting that Freon performs well across a wide range of exponents, we introduce Kaon, an absurd optimizer that replaces singular values with random noise. Despite lacking any coherent geometric structure, Kaon matches Muon's performance and retains classical convergence guarantees, proving that strict adherence to a precise geometry is practically irrelevant. Third, having shown that geometry is not the primary driver of performance, we demonstrate it is instead controlled by two local quantities: alignment and descent potential. Ultimately, each optimizer must tune its step size around these two quantities. While their dynamics are difficult to predict a-priori, evaluating them within a stochastic random feature model yields a precise insight: Muon succeeds not by tracking an ideal global geometry, but by guaranteeing step-size optimality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。