通过重复优化步加速训练，显著减少计算量和时间。
Fast Forwarding Low-Rank Training
- 交替使用常规优化与快速前进阶段，提升效率
- 相比标准SGD，FLOPs降低87%，训练时间减少81%
- 适合追求高效微调的NLP研究者与工程师
参数高效微调方法如低秩适应（LoRA）旨在降低预训练语言模型（LMs）微调的计算成本。在低秩设置基础上，我们提出一种更高效的优化策略：快速前进（Fast Forward），一种简单有效的加速大规模训练的方法。在快速前进阶段，我们重复最近一次优化器步骤，直到在极小验证集上损失不再下降。通过交替执行常规优化步骤与快速前进阶段，该方法在保持模型性能的前提下，相比标准SGD with Adam，实现高达87%的浮点运算量（FLOPs）减少和高达81%的训练时间节省。我们在多个模型和任务上验证了该方法的有效性，并分析了其适用时机与方式。
原文摘要 · Abstract (English)
Parameter efficient finetuning methods like low-rank adaptation (LoRA) aim to reduce the computational costs of finetuning pretrained Language Models (LMs). Enabled by these low-rank settings, we propose an even more efficient optimization strategy: Fast Forward, a simple and effective approach to accelerate large segments of training. In a Fast Forward stage, we repeat the most recent optimizer step until the loss stops improving on a tiny validation set. By alternating between regular optimization steps and Fast Forward stages, Fast Forward provides up to an 87\% reduction in FLOPs and up to an 81\% reduction in train time over standard SGD with Adam. We validate Fast Forward by finetuning various models on different tasks and demonstrate that it speeds up training without compromising model performance. Additionally, we analyze when and how to apply Fast Forward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。