提出SOAA优化器，用对角近似加速二阶优化，适合大模型训练。
Efficient Second-Order Neural Network Optimization via Adaptive Trust Region Methods
- 用对角化费雪信息矩阵，将复杂度从O(n²)降到O(n)
- 自适应调整信任区域大小，收敛更快更稳定
- 适合大语言模型等大规模深度学习任务
二阶优化方法通过利用曲率信息可实现更快收敛，但传统方法因大规模矩阵求逆和高内存需求而计算成本过高。尽管已开发出自适应信任域方法缓解此问题，其性能常受赫森矩阵利普希茨常数保守估计的限制。本文提出SecondOrderAdaptiveAdam（SOAA），通过使用对角化费雪信息矩阵近似，将计算复杂度从$O(n^{2})$降低至$O(n)$，使其适用于大规模深度学习模型，包括大语言模型（LLMs）。该算法结合自适应信任域机制，根据实际损失下降动态调整信任区域大小，兼顾收敛稳定性与计算效率。实验表明，在相似计算约束下，SOAA比一阶优化器如Adam收敛更快且更稳定。然而，费雪信息矩阵的对角近似可能难以捕捉梯度间的高阶交互，提示未来仍有改进空间。
原文摘要 · Abstract (English)
Second-order optimization methods offer notable advantages in training deep neural networks by utilizing curvature information to achieve faster convergence. However, traditional second-order techniques are computationally prohibitive, primarily due to the large matrix inversions and high memory demands they require. While adaptive trust-region methods have been developed to mitigate these issues, their performance is often hindered by conservative estimates of key parameters, such as the Lipschitz constant of the Hessian, resulting in suboptimal outcomes. In this paper, we introduce SecondOrderAdaptiveAdam (SOAA), a novel optimization algorithm designed to overcome these limitations. SOAA approximates the Fisher information matrix using a diagonal representation, reducing computational complexity from \(O(n^{2})\) to \(O(n)\), thereby making it suitable for large-scale deep learning models, including large language models (LLMs). Additionally, the algorithm integrates an adaptive trust-region mechanism that dynamically adjusts the trust region size based on observed loss reduction, ensuring both robust convergence and computational efficiency. We empirically demonstrate that SOAA achieves faster and more stable convergence compared to first-order optimizers, such as Adam, under similar computational constraints. However, the diagonal approximation of the Fisher information matrix may be less effective in capturing higher-order interactions between gradients, suggesting potential areas for further refinement and future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。