LionMuon交替使用两种优化器，用更低算力实现更强训练效果。
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
- 交替使用Lion和Muon更新，共享一个动量缓冲区
- 在124M~720M模型上均优于现有方法，降低算力且损失更低
- 适合追求高效训练的深度学习研究者与工程师
大规模优化中，更新步骤的性价比是成功的关键。基于符号的优化器如Lion或Signum每步开销低，而Muon的谱矩阵符号更新方向更强但每步代价高。本文提出LionMuon，保留Muon的有效性同时显著降低平均迭代成本，接近符号类方法。它以固定周期P交替执行Lion与Muon更新，并共享单一双指数移动平均动量缓冲区，优化器状态内存仅相当于Lion，为AdamW的一半。更简单的单动量变体SignMuon已超越纯Muon。当周期P=2时，LionMuon在所有测试数据集与架构上均优于Muon、Lion、Signum及AdamW，124M模型下实现更低验证损失与更少算力消耗；该优势在355M和720M模型上持续存在。理论上，我们证明了在重尾噪声下的紧致复杂度界，其由周期平均光滑性和噪声决定，介于Muon与Lion常数之间。这些界限可预测最优计算周期，以及LionMuon超越Muon与Lion的条件。
原文摘要 · Abstract (English)
In large-scale optimization, the cheapness and effectiveness of update steps are the most crucial factors for a successful optimizer. Sign-based optimizers like Lion or Signum produce cheap per-step updates, whereas Muon's spectral matrix-sign update gives a much stronger direction at a substantially higher per-step cost. In this work, we propose LionMuon, which retains the effectiveness of Muon steps while considerably cutting the averaged iteration cost, similar to sign-based methods. It alternates between Lion's and Muon's updates on a fixed period P, sharing a single dual-EMA momentum buffer between them. The optimizer state memory therefore matches Lion and is exactly half of AdamW's. A simpler single-EMA variant, SignMuon, by itself already outperforms pure Muon. At P = 2, LionMuon Pareto-dominates Muon, Lion, Signum, and AdamW on every dataset and architecture we tested at 124M model size, reaching lower validation loss at lower compute, and the same advantage persists at 355M and 720M scale. On the theory side, we prove sharp complexity bounds under heavy-tailed noise which are governed by period-averaged smoothness and noise that interpolate between Muon's and Lion's constants. These bounds predict the compute-optimal period and the conditions under which LionMuon outruns Muon and Lion. Code: https://github.com/brain-lab-research/lion-muon
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。