自适应调整内存与计算开销，让大模型训练更省显存更快
AdaFRUGAL: Adaptive Memory-Efficient Training with Dynamic Control
- 动态调节梯度分块比例和更新频率，自动优化资源使用
- 在英/越语数据集上显存减少超30%，训练速度提升25%以上
- 适合显存有限但需高效训练大模型的研究者与开发者
大型语言模型（LLMs）训练因优化器状态占用大量显存。FRUGAL框架通过梯度分块缓解此问题，但其静态超参数——子空间比例（ρ）和更新频率（T）——需耗时手动调优，限制了灵活性。本文提出AdaFRUGAL，引入两项动态控制：(i) ρ的线性衰减以逐步降低内存占用；(ii) 基于损失感知的T调度以减少计算开销。在大规模预训练（English C4、Vietnamese VietVault）和微调（GLUE）任务上的实验表明，AdaFRUGAL在保持与AdamW及静态FRUGAL相当性能的同时，显著降低GPU显存使用量与训练时间，为资源受限场景下的大模型训练提供更实用、自主的解决方案。
原文摘要 · Abstract (English)
Training Large Language Models (LLMs) is highly memory-intensive due to optimizer state overhead. The FRUGAL framework mitigates this with gradient splitting, but its static hyperparameters -- the subspace ratio ($ρ$) and update frequency ($T$) -- require costly manual tuning, limiting adaptability. We present AdaFRUGAL, which automates this process by introducing two dynamic controls: (i) a linear decay for $ρ$ to progressively reduce memory, and (ii) a loss-aware schedule for $T$ to lower computational overhead. Experiments across large-scale pre-training (English C4, Vietnamese VietVault) and fine-tuning (GLUE) demonstrate that AdaFRUGAL achieves a compelling trade-off. It maintains competitive performance against AdamW and static FRUGAL while significantly reducing both GPU memory and training time, offering a more practical, autonomous solution for resource-constrained LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。