GaLore 2通过低秩投影加速大模型训练，显著降低显存占用。
GaLore 2: Large-Scale LLM Pre-Training by Gradient Low-Rank Projection
- 用梯度低秩投影压缩内存，减少训练显存消耗
- 在5000亿词上从零训练Llama 7B，性能无损
- 兼容FSDP等先进并行策略，适合大规模预训练
大语言模型（LLMs）在自然语言理解与生成方面取得突破，但训练过程面临严重的内存瓶颈。GaLore，即梯度低秩投影，利用权重梯度的内在低秩特性，实现显著的内存节省且不牺牲性能。近期研究从低比特量化和高阶张量结构等方面扩展了GaLore，但仍存在如子空间更新中SVD计算开销大、与先进训练并行策略（如FSDP）集成困难等挑战。本文提出GaLore 2，一个高效可扩展的框架，解决了上述问题并融合最新进展。此外，我们通过在最多5000亿训练词上从零预训练Llama 7B，展示了GaLore 2的可扩展性，凸显其在真实大模型预训练场景中的潜力。
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized natural language understanding and generation but face significant memory bottlenecks during training. GaLore, Gradient Low-Rank Projection, addresses this issue by leveraging the inherent low-rank structure of weight gradients, enabling substantial memory savings without sacrificing performance. Recent works further extend GaLore from various aspects, including low-bit quantization and higher-order tensor structures. However, there are several remaining challenges for GaLore, such as the computational overhead of SVD for subspace updates and the integration with state-of-the-art training parallelization strategies (e.g., FSDP). In this paper, we present GaLore 2, an efficient and scalable GaLore framework that addresses these challenges and incorporates recent advancements. In addition, we demonstrate the scalability of GaLore 2 by pre-training Llama 7B from scratch using up to 500 billion training tokens, highlighting its potential impact on real LLM pre-training scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。