提出新优化器Mano,让大模型训练更快更省资源。
Mano: Restriking Manifold Optimization for LLM Training
- 将动量投影到参数流形切空间,约束在旋转斜交流形上。
- 在LLaMA和Qwen3上优于AdamW和Muon,且内存与计算更低。
- 适合追求高效训练的大模型研究者或工业部署场景。
尽管大语言模型(LLMs)在人工智能领域取得显著进展,但其训练所需的硬件与计算成本仍极为高昂。现有先进优化器中,AdamW依赖对角曲率估计并忽略结构特性,而Muon虽采用全局谱归一化,却损失了曲率信息。本文重新探索流形优化方法用于大模型训练,克服了传统流形优化在大规模模型中表现不佳的问题。通过创新性地将动量投影至模型参数的切空间,并在旋转斜交流形上施加约束,提出新型高效优化器Mano,首次弥合了流形优化与现代优化器之间的性能差距。在LLaMA与Qwen3模型上的大量实验表明,Mano在保持更低内存消耗和计算复杂度的前提下,持续显著超越AdamW与Muon,展现出更优的空间与时间效率权衡,拓展了优化器的帕累托前沿。
原文摘要 · Abstract (English)
While large language models (LLMs) have emerged as a significant advancement in artificial intelligence, the hardware and computational costs for training LLMs are also significantly burdensome. Among the state-of-the-art optimizers, AdamW relies on diagonal curvature estimates and ignores structural properties, while Muon applies global spectral normalization at the expense of losing curvature information. In this study, we restriked manifold optimization methods for training LLMs, which may address both optimizers' limitations, while conventional manifold optimization methods have been largely overlooked due to the poor performance in large-scale model optimization. By innovatively projecting the momentum onto the tangent space of model parameters and constraining it on a rotational Oblique manifold, we propose a novel, powerful, and efficient optimizer **Mano** that is the first to bridge the performance gap between manifold optimization and modern optimizers. Extensive experiments on the LLaMA and Qwen3 models demonstrate that Mano consistently and significantly outperforms AdamW and Muon even with less memory consumption and computational complexity, respectively, suggesting an expanded Pareto frontier in terms of space and time efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。