用尼斯特罗姆近似加速梯度下降，兼顾精度与效率。
NysAct: A Scalable Preconditioned Gradient Descent using Nystrom Approximation
- 用特征值平移的尼斯特罗姆法近似激活协方差矩阵作预条件
- 测试准确率优于一阶和二阶方法，计算开销远低于传统二阶方法
- 适合需要高效高精度训练的大模型场景
自适应梯度方法计算高效且收敛快，但泛化性能较差；而二阶方法虽提升收敛与泛化，却常伴随高昂的计算与内存开销。本文提出NysAct，一种可扩展的一阶梯度预条件方法，平衡了当前最优一阶与二阶优化方法的性能。NysAct采用特征值平移的尼斯特罗姆方法近似激活协方差矩阵作为预条件矩阵，显著降低时间和内存复杂度，对测试准确率影响极小。实验表明，NysAct在测试准确率上优于一阶与二阶方法，且相比现有二阶方法所需计算资源大幅减少。代码已公开于https://github.com/hseung88/nysact。
原文摘要 · Abstract (English)
Adaptive gradient methods are computationally efficient and converge quickly, but they often suffer from poor generalization. In contrast, second-order methods enhance convergence and generalization but typically incur high computational and memory costs. In this work, we introduce NysAct, a scalable first-order gradient preconditioning method that strikes a balance between state-of-the-art first-order and second-order optimization methods. NysAct leverages an eigenvalue-shifted Nystrom method to approximate the activation covariance matrix, which is used as a preconditioning matrix, significantly reducing time and memory complexities with minimal impact on test accuracy. Our experiments show that NysAct not only achieves improved test accuracy compared to both first-order and second-order methods but also demands considerably less computational resources than existing second-order methods. Code is available at https://github.com/hseung88/nysact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。