用低秩动态更新提升大模型优化速度
Dynamic Low-rank Approximation of Full-Matrix Preconditioner for Training Generalized Linear Models
- 通过低秩矩阵动态逼近全矩阵预条件器，实现高效优化
- 在秩5以下时收敛速度优于或等同于传统对角方法
- 适合大规模模型的自适应优化，兼顾性能与资源开销
自适应梯度方法如Adagrad及其变体广泛应用于大规模优化，但其使用对角预条件矩阵限制了参数相关性的捕捉能力。全矩阵自适应方法可近似精确的海森矩阵，建模参数相关性，可能加快收敛速度。然而，其计算和内存开销通常难以在大规模模型中承受。为此，我们提出AdaGram优化器，实现高效的全矩阵自适应梯度更新。为降低内存和计算开销，每轮迭代采用快速对称分解计算预条件更新方向，并利用矩阵积分方法沿优化轨迹保持预条件器的低秩结构。标准机器学习任务上的数值实验表明，当使用秩五及以下的近似时，AdaGram的收敛速度优于或匹配对角自适应优化器的表现。这证明了AdaGram在大规模模型自适应优化中的可扩展潜力。
原文摘要 · Abstract (English)
Adaptive gradient methods like Adagrad and its variants are widespread in large-scale optimization. However, their use of diagonal preconditioning matrices limits the ability to capture parameter correlations. Full-matrix adaptive methods, approximating the exact Hessian, can model these correlations and may enable faster convergence. At the same time, their computational and memory costs are often prohibitive for large-scale models. To address this limitation, we propose AdaGram, an optimizer that enables efficient full-matrix adaptive gradient updates. To reduce memory and computational overhead, we utilize fast symmetric factorization for computing the preconditioned update direction at each iteration. Additionally, we maintain the low-rank structure of a preconditioner along the optimization trajectory using matrix integrator methods. Numerical experiments on standard machine learning tasks show that AdaGram converges faster or matches the performance of diagonal adaptive optimizers when using rank five and smaller rank approximations. This demonstrates AdaGram's potential as a scalable solution for adaptive optimization in large models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。