针对深度模型优化结构特性，提出自适应梯度优化新算法
ASGO: Adaptive Structured Gradient Optimization
- 基于参数矩阵结构设计自适应预处理机制
- 理论上证明收敛速度优于现有结构化优化方法
- 适合语言模型训练，可利用梯度低秩与海森块对角特性
深度神经网络训练本质上是具有结构的优化问题，因参数天然以矩阵和张量形式存在。研究发现，此类问题中梯度通常为低秩，海森矩阵近似块对角。这些结构性质对高效优化算法设计至关重要，但当前主流优化器（如Adam）并未加以利用。本文提出新型优化算法ASGO，通过使用基于结构化梯度自适应更新的预处理矩阵，充分挖掘上述结构特性。理论分析表明，ASGO的收敛速率优于现有结构化梯度方法。进一步论证其可受益于低秩梯度与块对角海森矩阵。还讨论了实用改进方案，并在语言模型任务上实证验证了ASGO的有效性。代码已开源：https://github.com/infinity-stars/ASGO。
原文摘要 · Abstract (English)
Training deep neural networks is a structured optimization problem, because the parameters are naturally represented by matrices and tensors rather than by vectors. Under this structural representation, it has been widely observed that gradients are low-rank and Hessians are approximately block diagonal. These structured properties are crucial for designing efficient optimization algorithms, but are not utilized by many current popular optimizers like Adam. In this paper, we present a novel optimization algorithm ASGO that capitalizes on these properties by employing a preconditioner that is adaptively updated using structured gradients. By a fine-grained theoretical analysis, ASGO is proven to achieve superior convergence rates compared to existing structured gradient methods. Based on this convergence theory, we further demonstrate that ASGO can benefit from low-rank gradients and block diagonal Hessians. We also discuss practical modifications of ASGO and empirically verify ASGO's effectiveness on language model tasks. Code is available at https://github.com/infinity-stars/ASGO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。