LDAdam通过低维梯度统计实现高效自适应优化，内存占用仅为模型大小的一小部分。
LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics
- 在低维子空间内进行自适应更新，同时保持对全参数空间的探索
- 相比传统优化器，内存开销降低至模型大小的1/10以下，支持大模型训练
- 适合资源受限场景下的大语言模型预训练与微调，尤其适合显存不足的情况
我们提出LDAdam，一种面向大规模模型训练的内存高效的优化器。该方法在低维子空间中执行自适应优化步骤，同时在整个参数空间中持续探索。这一策略使优化器的内存占用远低于模型规模。LDAdam采用新的投影感知更新规则，支持在不同子空间间切换，能够准确估计投影梯度的统计特性。为缓解低秩投影带来的误差，引入一种广义误差反馈机制，显式建模梯度与优化器状态压缩的影响。在标准假设下证明了其收敛性，并验证了其在语言模型预训练与微调中的高精度与高效性。代码已开源。
原文摘要 · Abstract (English)
We introduce LDAdam, a memory-efficient optimizer for training large models, that performs adaptive optimization steps within lower dimensional subspaces, while consistently exploring the full parameter space during training. This strategy keeps the optimizer's memory footprint to a fraction of the model size. LDAdam relies on a new projection-aware update rule for the optimizer states that allows for transitioning between subspaces, i.e., estimation of the statistics of the projected gradients. To mitigate the errors due to low-rank projection, LDAdam integrates a new generalized error feedback mechanism, which explicitly accounts for both gradient and optimizer state compression. We prove the convergence of LDAdam under standard assumptions, and show that LDAdam allows for accurate and efficient fine-tuning and pre-training of language models. Code is available at https://github.com/IST-DASLab/LDAdam
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。