通过细粒度梯度投影提升大模型训练内存效率
Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradients
- 引入可调粒度的低秩梯度投影框架，突破传统固定行级投影限制
- 相同内存预算下，细粒度投影使训练更稳定且效率更高
- 提出ProjFactor优化器，大幅降低显存占用，适合资源受限场景
基于低秩适配器（LoRA）的成功，低秩梯度投影（LoRP）已成为内存高效微调的有前景方案。然而现有方法通常将梯度矩阵的每一行作为默认投影单元，未充分探索投影粒度的影响。本文提出新框架VLoRP，通过引入额外自由度控制内存效率与性能之间的权衡，超越了单纯依赖秩超参数的局限。系统研究了投影粒度的影响，发现固定内存预算下，更细粒度的投影能显著提升训练稳定性和效率。针对VLoRP优化，提出ProjFactor自适应内存高效优化器，在梯度累积情况下仍保持低内存开销并取得竞争力表现。同时提供理论分析，证明在SGD和ProjFactor下优化轨迹的下降性与收敛性。大量实验验证了结论，涵盖常识推理、MMLU和GSM8K等任务。
原文摘要 · Abstract (English)
Building upon the success of low-rank adapter (LoRA), low-rank gradient projection (LoRP) has emerged as a promising solution for memory-efficient fine-tuning. However, existing LoRP methods typically treat each row of the gradient matrix as the default projection unit, leaving the role of projection granularity underexplored. In this work, we propose a novel framework, VLoRP, that extends low-rank gradient projection by introducing an additional degree of freedom for controlling the trade-off between memory efficiency and performance, beyond the rank hyper-parameter. Through this framework, we systematically explore the impact of projection granularity, demonstrating that finer-grained projections lead to enhanced stability and efficiency even under a fixed memory budget. Regarding the optimization for VLoRP, we present ProjFactor, an adaptive memory-efficient optimizer, that significantly reduces memory requirement while ensuring competitive performance, even in the presence of gradient accumulation. Additionally, we provide a theoretical analysis of VLoRP, demonstrating the descent and convergence of its optimization trajectory under both SGD and ProjFactor. Extensive experiments are conducted to validate our findings, covering tasks such as commonsense reasoning, MMLU, and GSM8K.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。