Alada用交替更新降低矩阵优化内存开销，适合大模型训练。
Alada: Alternating Adaptation of Momentum Method for Memory-Efficient Matrix Optimization
- 通过交替更新秩一分解因子，低内存估算梯度二阶矩。
- 相比Adam内存开销接近线性下降，训练大模型更稳定。
- 无需额外内存，可直接用于张量变量优化，适合资源受限场景。
本文提出Alada，一种面向大规模矩阵的自适应动量优化方法。Alada采用秩一分解方式估计梯度的二阶矩，通过交替更新分解因子以最小化估计误差，实现近似子线性的内存开销，并可自然扩展至张量形状变量的优化。同时，Alada引入一阶矩估计规则，在不增加内存开销的前提下增强算法鲁棒性。理论分析表明，Alada性能与传统方法如Adam相当。在多个自然语言处理任务上的数值实验验证了其在降低内存消耗和提升大模型训练稳定性方面优于Adam及其变体。
原文摘要 · Abstract (English)
This work proposes Alada, an adaptive momentum method for stochastic optimization over large-scale matrices. Alada employs a rank-one factorization approach to estimate the second moment of gradients, where factors are updated alternatively to minimize the estimation error. Alada achieves sublinear memory overheads and can be readily extended to optimizing tensor-shaped variables.We also equip Alada with a first moment estimation rule, which enhances the algorithm's robustness without incurring additional memory overheads. The theoretical performance of Alada aligns with that of traditional methods such as Adam. Numerical studies conducted on several natural language processing tasks demonstrate the reduction in memory overheads and the robustness in training large models relative to Adam and its variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。