POET-X让大模型用单块显卡高效训练，内存占用大幅降低
POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
- 通过优化正交变换方式减少计算开销
- 在单张H100上成功训练十亿参数大模型
- 适合资源有限但需稳定训练大模型的研究者
大规模语言模型（LLM）的高效稳定训练仍是现代机器学习系统的核心挑战。为应对这一问题，已有研究提出重参数化正交等价训练（POET），通过正交等价变换优化每层权重矩阵，保持谱特性并提升训练稳定性。然而，原始POET实现因大量矩阵乘法导致高内存消耗和计算开销。为此，本文提出可扩展且内存高效的变体POET-X，以显著降低正交等价变换的计算成本。实验表明，POET-X在保持POET的泛化性与稳定性的同时，大幅提升吞吐量和内存效率。在相同设置下，POET-X可在单张Nvidia H100 GPU上完成十亿参数级大模型的预训练，而标准优化器如AdamW则因显存不足无法运行。
原文摘要 · Abstract (English)
Efficient and stable training of large language models (LLMs) remains a core challenge in modern machine learning systems. To address this challenge, Reparameterized Orthogonal Equivalence Training (POET), a spectrum-preserving framework that optimizes each weight matrix through orthogonal equivalence transformation, has been proposed. Although POET provides strong training stability, its original implementation incurs high memory consumption and computational overhead due to intensive matrix multiplications. To overcome these limitations, we introduce POET-X, a scalable and memory-efficient variant that performs orthogonal equivalence transformations with significantly reduced computational cost. POET-X maintains the generalization and stability benefits of POET while achieving substantial improvements in throughput and memory efficiency. In our experiments, POET-X enables the pretraining of billion-parameter LLMs on a single Nvidia H100 GPU, and in contrast, standard optimizers such as AdamW run out of memory under the same settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。