提出Stacey优化算法，用非欧几何提升深度学习训练速度与精度
Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization
- 基于插值的对偶迭代序列，实现非欧光滑优化的加速梯度下降
- 在图像分类和大模型预训练中，收敛更快且最终准确率更高
- 验证不同p值效果，证明非欧方法优于传统欧氏方法
尽管SGD、AdamW和Lion等主流优化方法依赖于ℓ₂或ℓ∞范数的最速下降更新，但在现代深度网络训练中仍存在处理非欧几里得结构的显著缺口。本文提出一种新的加速ℓₚ最速下降算法Stacey，通过插值的原-对偶迭代序列，有效应对非欧光滑优化任务。除了为算法提供新颖的理论保证外，我们在图像分类和语言模型（LLM）预训练任务上对主流方法进行了实证比较，结果表明该方法具备更快的收敛速度和更高的最终精度。我们还评估了不同p值在多种模型和数据集上的表现，凸显非欧方法相较于标准欧氏方法的重要性和高效性。代码见https://github.com/xinyuluo8561/Stacey。
原文摘要 · Abstract (English)
While popular optimization methods such as SGD, AdamW, and Lion depend on steepest descent updates in either $\ell_2$ or $\ell_\infty$ norms, there remains a critical gap in handling the non-Euclidean structure observed in modern deep networks training. In this work, we address this need by introducing a new accelerated $\ell_p$ steepest descent algorithm, called Stacey, which uses interpolated primal-dual iterate sequences to effectively navigate non-Euclidean smooth optimization tasks. In addition to providing novel theoretical guarantees for the foundations of our algorithm, we empirically compare our approach against these popular methods on tasks including image classification and language model (LLM) pretraining, demonstrating both faster convergence and higher final accuracy. We further evaluate different values of $p$ across various models and datasets, underscoring the importance and efficiency of non-Euclidean approaches over standard Euclidean methods. Code can be found at https://github.com/xinyuluo8561/Stacey .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。