分析signSGD在线性回归中的加速机制，揭示其在特定条件下优于SGD的计算优化规律。
Scaling Laws of SignSGD in Linear Regression: When Does It Outperform SGD?
- 基于幂律随机特征模型，推导signSGD的种群风险表达式。
- 发现噪声重塑效应使signSGD在噪声主导时计算最优斜率更陡，收敛更快。
- 验证WSD学习率调度可进一步降低噪声，适合特征衰减快、目标衰减慢场景。
我们在一个包含特征与目标衰减的幂律随机特征（PLRF）模型下研究signSGD的缩放规律。分析了在高斯投影特征上单遍训练的线性模型使用signSGD的种群风险，将其表示为模型规模、训练步数、学习率以及特征和目标衰减参数的函数。相较于Paquette等（2024）对SGD风险的分析，我们识别出signSGD独有的漂移归一化效应和噪声重塑效应。在最优学习率下获得计算最优缩放规律。结果表明，在噪声主导的区域，噪声重塑效应可使signSGD的计算最优斜率比SGD更陡。最后发现，当特征衰减快而目标衰减慢时，广泛使用的预热-稳定-衰减（WSD）调度能进一步抑制噪声并提升计算最优斜率。
原文摘要 · Abstract (English)
We study scaling laws of signSGD under a power-law random features (PLRF) model that accounts for both feature and target decay. We analyze the population risk of a linear model trained with one-pass signSGD on Gaussian-sketched features. We express the risk as a function of model size, training steps, learning rate, and the feature and target decay parameters. Comparing against the SGD risk analyzed by Paquette et al. (2024), we identify a drift-normalization effect and a noise-reshaping effect unique to signSGD. We then obtain compute-optimal scaling laws under the optimal choice of learning rate. Our analysis shows that the noise-reshaping effect can make the compute-optimal slope of signSGD steeper than that of SGD in regimes where noise is dominant. Finally, we observe that the widely used warmup-stable-decay (WSD) schedule further reduces the noise term and sharpens the compute-optimal slope, when feature decay is fast but target decay is slow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。