Pion通过正交变换保持模型权重谱不变，提升训练稳定性。
Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation
- 用左右正交变换更新权重矩阵，保持奇异值不变
- 在大语言模型预训练与微调中表现稳定且效果不逊于传统优化器
- 适合追求训练稳定性的大规模模型优化场景
我们提出 Pion，一种基于正交等价变换的大语言模型训练谱保优优化器。与 Adam、Muon 等加性优化器不同，Pion 通过左、右正交变换更新每个权重矩阵，全程保持其奇异值不变。这使得优化过程在调节权重矩阵几何结构的同时，固定其谱范数。我们推导了 Pion 的更新规则，系统分析了设计选择及其收敛行为与若干关键性质。实验结果表明，Pion 在大语言模型预训练与微调任务中提供了稳定且具有竞争力的优化替代方案。
原文摘要 · Abstract (English)
We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。