通过追踪梯度子空间，实现大模型训练的高效低耗。
SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training
- 用格拉斯曼几何追踪梯度子空间，动态适应优化器状态变化。
- 相比现有最优方法，预训练耗时减少65%，微调时间减少36%。
- 适合资源受限场景下的大模型训练，兼顾速度、内存与性能。
大规模语言模型（LLMs）训练因参数量巨大和优化器状态开销而资源消耗高。尽管已有研究致力于降低内存占用，但常在内存效率、训练时间与模型性能间权衡。真正的大模型普及需三者同步提升。为此，我们提出SubTrack++，结合格拉斯曼梯度子空间追踪与投影感知优化器，使Adam的内部统计量能适应子空间变化；同时引入恢复缩放技术，弥补低秩投影导致的信息损失。该方法利用格拉斯曼几何实现当前最优收敛性，在保持相同内存开销的前提下，相比现有最优方法，预训练墙时长最多降低65%，微调时间减少36%。
原文摘要 · Abstract (English)
Training large language models (LLMs) is highly resource-intensive due to their massive number of parameters and the overhead of optimizer states. While recent work has aimed to reduce memory consumption, such efforts often entail trade-offs among memory efficiency, training time, and model performance. Yet, true democratization of LLMs requires simultaneous progress across all three dimensions. To this end, we propose SubTrack++ that leverages Grassmannian gradient subspace tracking combined with projection-aware optimizers, enabling Adam's internal statistics to adapt to subspace changes. Additionally, employing recovery scaling, a technique that restores information lost through low-rank projections, further enhances model performance. Our method demonstrates SOTA convergence by exploiting Grassmannian geometry, reducing pre-training wall-time by up to 65% and fine-tuning time by 36% compared to existing SOTA methods, while maintaining the same memory footprint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。