GaLore+通过跨头投影加速大模型微调，速度提升4倍。
GaLore$+$: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection
- 用跨头低秩投影替代传统SVD，减少计算耗时。
- 结合随机子空间迭代，使SVD更快，训练速度提升4倍。
- 引入稀疏残差编码，降低低秩近似误差，适合资源受限场景。
近期的低秩训练方法（如GaLore）显著降低了大语言模型（LLMs）优化所需的内存。然而，这些方法常因低秩投影估计耗时过长而受限，尤其在GaLore中，奇异值分解（SVD）占总训练时间超过80%。为此，我们提出GaLore+，采用跨头低秩投影以减少多头注意力中低秩投影估计的时间开销。同时，使用随机子空间迭代实现快速SVD。为进一步提升性能，我们提出稀疏编码残差，以降低优化器一阶和二阶矩及权重更新中低秩近似的误差。我们在算术推理和自然语言生成数据集上评估GaLore+，实验表明其在保持优异性能的同时，相比原始GaLore实现了约4倍的微调速度提升。
原文摘要 · Abstract (English)
Recent low-rank training methods, such as GaLore, have significantly reduced the memory required to optimize large language models (LLMs). However, these methods often suffer from time-consuming low-rank projection estimations. In particular, the singular value decomposition (SVD) in GaLore can consume more than 80\% of the total training time. To address this issue, we propose GaLore$+$, which uses cross-head low-rank projection to reduce the substantial time consumption in estimating low-rank projections for multi-head attention. In addition, we employ randomized subspace iteration to achieve fast SVD. To further enhance performance, we propose sparsely coded residuals to reduce the errors caused by low-rank approximation on the first- and second-order moments of the optimizers and weight updates. We evaluate GaLore$+$ on arithmetic reasoning and natural language generation datasets. Our experiments demonstrate that GaLore$+$ delivers superior performance while achieving approximately $4\times$ fine-tuning speed compared to vanilla GaLore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。