通过正交变换重参数化神经元，提升大模型训练稳定性与泛化能力。
Reparameterized LLM Training via Orthogonal Equivalence Transformation
- 用两个可学习正交矩阵加固定随机矩阵重参数化神经元
- 保持权重矩阵谱特性，优化目标更稳定，泛化性能提升
- 算法高效可扩展，适合大规模语言模型训练
尽管大语言模型（LLMs）正推动人工智能快速发展，但有效且可靠地训练这些大型模型仍是该领域最重要的挑战之一。为此，我们提出POET——一种基于正交等价变换的新型重参数化训练算法，用于优化神经元。具体而言，POET通过两个可学习的正交矩阵和一个固定随机权重矩阵对每个神经元进行重参数化。由于其能保证权重矩阵谱特性的可证明保留性，POET可在优化目标函数时实现更稳定的收敛，并提升模型泛化能力。我们进一步设计了高效的近似方法，使POET在训练大规模神经网络时具备灵活性与可扩展性。大量实验验证了POET在训练大语言模型中的有效性与可扩展性。
原文摘要 · Abstract (English)
While large language models (LLMs) are driving the rapid advancement of artificial intelligence, effectively and reliably training these large models remains one of the field's most significant challenges. To address this challenge, we propose POET, a novel reParameterized training algorithm that uses Orthogonal Equivalence Transformation to optimize neurons. Specifically, POET reparameterizes each neuron with two learnable orthogonal matrices and a fixed random weight matrix. Because of its provable preservation of spectral properties of weight matrices, POET can stably optimize the objective function with improved generalization. We further develop efficient approximations that make POET flexible and scalable for training large-scale neural networks. Extensive experiments validate the effectiveness and scalability of POET in training LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。