DANA算法通过自适应调整动量，显著优于传统SGD-M。
Dimension-adapted Momentum Outscales SGD
- 根据模型规模和数据复杂度动态调整动量超参
- 在多种复杂度下损失下降速度更快，计算效率更高
- 理论与实验验证，适合大规模高维任务
我们研究了在幂律随机特征模型上，小批量随机动量算法的缩放规律，该模型由数据复杂度、目标复杂度和模型大小参数化。当使用随机动量算法训练时，分析揭示了四种由数据-目标复杂度变化决定的损失曲线形态。虽然传统的带动量随机梯度下降（SGD-M）的缩放指数与SGD相同，但维度自适应的Nesterov加速（DANA）通过基于模型大小和数据复杂度缩放动量超参，提升了这些指数。这种超越缩放现象也改善了计算最优的缩放行为，在广泛的复杂度范围内，传统方法无法达到此效果。大量高维合成二次型实验验证了理论预测，大规模文本实验中使用LSTM的测试表明，DANA在实际场景中依然优于SGD的损失指数。
原文摘要 · Abstract (English)
We investigate scaling laws for stochastic momentum algorithms with small batch on the power law random features model, parameterized by data complexity, target complexity, and model size. When trained with a stochastic momentum algorithm, our analysis reveals four distinct loss curve shapes determined by varying data-target complexities. While traditional stochastic gradient descent with momentum (SGD-M) yields identical scaling law exponents to SGD, dimension-adapted Nesterov acceleration (DANA) improves these exponents by scaling momentum hyperparameters based on model size and data complexity. This outscaling phenomenon, which also improves compute-optimal scaling behavior, is achieved by DANA across a broad range of data and target complexities, while traditional methods fall short. Extensive experiments on high-dimensional synthetic quadratics validate our theoretical predictions and large-scale text experiments with LSTMs show DANA's improved loss exponents over SGD hold in a practical setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。