用简单线性代数重推SGD在线性回归中的收敛分析。
A Simplified Analysis of SGD for Linear Regression with Weight Averaging
- 基于基础线性代数，避免复杂矩阵运算
- 复现了已有研究的偏差与方差边界
- 为优化算法改进提供清晰分析框架
理论上理解过参数模型中随机梯度下降（SGD）的性能，催生了多种如今广泛应用的优化算法。近期工作~\citet{zou2021benign} 基于风险的偏差-方差分解，给出了使用固定学习率的线性回归中SGD的精确收敛速率，涵盖尾部迭代平均与否的情况。本文通过简单的线性代数工具，重新推导出相同的偏差与方差界，避免了对半正定（PSD）矩阵算子的复杂操作。我们认为该分析更简洁易懂，有助于进一步研究小批量训练和学习率调度，从而提升实际模型的训练效果。
原文摘要 · Abstract (English)
Theoretically understanding stochastic gradient descent (SGD) in overparameterized models has led to the development of several optimization algorithms that are widely used in practice today. Recent work by~\citet{zou2021benign} provides sharp rates for SGD optimization in linear regression using constant learning rate, both with and without tail iterate averaging, based on a bias-variance decomposition of the risk. In our work, we provide a simplified analysis recovering the same bias and variance bounds provided in~\citep{zou2021benign} based on simple linear algebra tools, bypassing the requirement to manipulate operators on positive semi-definite (PSD) matrices. We believe our work makes the analysis of SGD on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling, leading to improvements in the training of realistic models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。