提出可保证收敛的低秩优化方法，解决大模型训练中的稳定性问题
Subspace Optimization for Large Language Models with Convergence Guarantees
- 设计基于梯度随机低秩投影的新算法GoLore，提升训练稳定性
- 理论证明在标准小批量下仍能收敛，突破原有方法局限
- 适用于追求高效且可靠训练的大模型研究者
子空间优化算法（如GaLore）因其内存效率高，被广泛用于大语言模型的预训练与微调。然而，其在随机设置下的收敛性尚未明确。本文揭示GaLore并不总能收敛至最优解，并提供显式反例。进一步分析表明，当使用足够大的小批量或梯度噪声各向同性时，其可收敛。更关键的是，我们提出GoLore（Gradient random Low-rank projection），一种新型变体，在典型随机设置下即使使用标准小批量也具备可证明的收敛性。我们的收敛分析可自然推广至其他子空间优化算法。最后，通过实验证实了理论结果并全面测试了所提机制。代码已公开于https://github.com/pkumelon/Golore。
原文摘要 · Abstract (English)
Subspace optimization algorithms, such as GaLore (Zhao et al., 2024), have gained attention for pre-training and fine-tuning large language models (LLMs) due to their memory efficiency. However, their convergence guarantees remain unclear, particularly in stochastic settings. In this paper, we reveal that GaLore does not always converge to the optimal solution and provide an explicit counterexample to support this finding. We further explore the conditions under which GaLore achieves convergence, showing that it does so when either (i) a sufficiently large mini-batch size is used or (ii) the gradient noise is isotropic. More significantly, we introduce GoLore (Gradient random Low-rank projection), a novel variant of GaLore that provably converges in typical stochastic settings, even with standard batch sizes. Our convergence analysis extends naturally to other subspace optimization algorithms. Finally, we empirically validate our theoretical results and thoroughly test the proposed mechanisms. Codes are available at https://github.com/pkumelon/Golore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。