在长尾分布下，符号下降法比标准梯度下降更快收敛。
Provable Benefit of Sign Descent: A Minimal Model Under Heavy-Tailed Class Imbalance
- 从数据分布出发，证明符号下降在坐标级优化中更优。
- 长尾类别不平衡时，符号下降收敛速度显著快于归一化GD。
- 适用于理解大模型训练中自适应优化器的优势。
自适应优化方法（如Adam）在大模型预训练中表现远超梯度下降（GD）。尽管已有研究通过损失函数的平滑性假设解释自适应算法的优势，但这些假设在语言建模任务中的具体体现仍不明确。本文从数据分布特性——长尾类别不平衡出发，构建一个最小且具代表性的下一词预测模型，严格证明：在长尾分布下，坐标级算法（如符号下降，即ℓ∞范数下的最速下降）相比归一化梯度下降（ℓ2范数下的最速下降）具有更快的收敛速度。
原文摘要 · Abstract (English)
Adaptive optimization methods (such as Adam) play a major role in LLM pretraining, significantly outperforming Gradient Descent (GD). Recent studies have proposed new smoothness assumptions on the loss function to explain the advantages of adaptive algorithms with structured preconditioners, e.g., coordinate-wise or layer-wise, and steepest descent methods w.r.t. non-euclidean norms, e.g., $\ell_\infty$ norm or spectral norm, over GD. However, it remains unclear how these smoothness assumptions manifest in language modelling tasks. In this work, we aim to analyze the benefit of $\ell_\infty$-norm descent (a.k.a. sign descent) directly from properties of the data distribution, namely, heavy-tailed class imbalance. We propose a minimal yet representative setting of next-token prediction, where we can provably show faster convergence of coordinate-wise algorithms such as Sign descent (steepest descent w.r.t. $\ell_\infty$ norm) over normalized GD (steepest descent w.r.t. to $\ell_2$ norm) in the presence of heavy tail class imbalance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。