Lotus通过随机低秩投影提升大模型训练效率，兼顾速度与内存消耗。
Lotus: Efficient LLM Training by Randomized Low-Rank Gradient Projection with Adaptive Subspace Switching
- 用新准则动态切换低秩梯度子空间，避免耗时的SVD计算
- 训练时间减少30%，梯度和优化器状态内存降40%
- 适合追求高效训练的大模型研究者和工程团队
大规模模型训练效率通常由内存消耗、训练时间和模型性能衡量。现有方法在三者间常存在权衡，优化一项往往损害其他。尽管GaLore通过低秩子空间更新梯度实现内存高效，但因对梯度进行奇异值分解（SVD）导致训练时间显著增加。本文提出Lotus，通过简化投影过程解决此权衡问题。我们设计了一种量化单位梯度位移的准则，实现低秩梯度子空间间的高效切换。实验表明，Lotus是目前最高效的训练方法，在预训练和微调任务中均优于基线，训练时间减少30%，梯度与优化器状态内存降低40%。
原文摘要 · Abstract (English)
Training efficiency in large-scale models is typically assessed through memory consumption, training time, and model performance. Current methods often exhibit trade-offs among these metrics, as optimizing one generally degrades at least one of the others. Addressing this trade-off remains a central challenge in algorithm design. While GaLore enables memory-efficient training by updating gradients in a low-rank subspace, it incurs a comparable extra training time cost due to the Singular Value Decomposition(SVD) process on gradients. In this paper, we propose Lotus, a method that resolves this trade-off by simply modifying the projection process. We propose a criterion that quantifies the displacement of the unit gradient to enable efficient transitions between low-rank gradient subspaces. Experimental results indicate that Lotus is the most efficient method, achieving a 30% reduction in training time and a 40% decrease in memory consumption for gradient and optimizer states. Additionally, it outperforms the baseline method in both pre-training and fine-tuning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。