通过梯度加权归一化与低秩投影，实现高效大模型训练。
Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
- 梯度加权归一化改善优化收敛性，低秩近似减少内存占用。
- 8位精度下优化器内存减少89.5%，可在消费级显卡上训练70亿参数模型。
- 在GLUE任务上优于LoRA，无需额外推理开销。
大型语言模型（LLMs）在各类任务中表现出色，但其对计算资源的高需求带来挑战，尤其体现在下游任务中广泛使用全量微调时。为应对这一问题，参数高效微调（PEFT）方法应运而生，但通常性能低于全量微调且内存效率不足。本文提出梯度加权归一化低秩投影（GradNormLoRP），在保持接近全量微调性能的同时，显著提升参数与内存效率。该方法通过归一化权重矩阵改善梯度条件，促进优化过程收敛；同时对权重和梯度矩阵进行低秩近似，大幅降低训练期间内存消耗。大量实验表明，8位精度下的GradNormLoRP可使优化器内存使用减少高达89.5%，支持在如NVIDIA RTX 4090等消费级显卡上预训练70亿参数的LLaMA模型，且不增加推理成本。此外，在相同秩为8的条件下，对RoBERTa模型在全部GLUE任务上的微调表现优于现有低秩方法，平均得分达80.65，超过LoRA的79.23。这些结果证明GradNormLoRP是高效大模型预训练与微调的有力候选方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable performance across various tasks, but the escalating demands on computational resources pose significant challenges, particularly in the extensive utilization of full fine-tuning for downstream tasks. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed, but they often underperform compared to full fine-tuning and struggle with memory efficiency. In this work, we introduce Gradient Weight-Normalized Low-Rank Projection (GradNormLoRP), a novel approach that enhances both parameter and memory efficiency while maintaining comparable performance to full fine-tuning. GradNormLoRP normalizes the weight matrix to improve gradient conditioning, facilitating better convergence during optimization. Additionally, it applies low-rank approximations to the weight and gradient matrices, significantly reducing memory usage during training. Extensive experiments demonstrate that our 8-bit GradNormLoRP reduces optimizer memory usage by up to 89.5% and enables the pre-training of large LLMs, such as LLaMA 7B, on consumer-level GPUs like the NVIDIA RTX 4090, without additional inference costs. Moreover, GradNormLoRP outperforms existing low-rank methods in fine-tuning tasks. For instance, when fine-tuning the RoBERTa model on all GLUE tasks with a rank of 8, GradNormLoRP achieves an average score of 80.65, surpassing LoRA's score of 79.23. These results underscore GradNormLoRP as a promising alternative for efficient LLM pre-training and fine-tuning. Source code: https://github.com/Jhhuangkay/Gradient-Weight-normalized-Low-rank-Projection-for-Efficient-LLM-Training
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。