提出新型优化方法，提升深度学习收敛速度与稳定性。
Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness
- 融合最速下降与条件梯度，实现更优的下降性质。
- 随机情形下达到最优 $O(n^{-1/4})$ 收敛率。
- 适用于图像分类与语言建模，代码开源可复现。
本文提出一种混合非欧几里得优化方法，通过结合最速下降与条件梯度方法，推广了梯度范数裁剪。该方法在广义的 $(L_0, L_1)$-光滑性假设下建立了下降性质。通过关联 Frank-Wolfe 短步，以合理方式引入权重衰减。在随机情形下，借助基于动量的梯度估计器，实现了阶最优的 $O(n^{-1/4})$ 收敛率。讨论了其在深度学习中的实例化方式，命名为 Clipped Scion，并在图像分类与语言建模任务上验证了其有效性。代码已公开于 https://github.com/LIONS-EPFL/ClippedScion。
原文摘要 · Abstract (English)
This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ($L_0$,$L_1$)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal $O(n^{-1/4})$ convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。