噪声训练让对角线网络快速平衡层间参数，提升泛化性能。
Training Diagonal Linear Networks with Stochastic Sharpness-Aware Minimization
- 用随机噪声模拟尖锐感知优化，引导参数向低敏感解收敛。
- 噪声强度控制参数收缩与阈值效应，影响层间平衡速度。
- 适用于理解深度线性模型的泛化机制，适合研究优化动力学者。
我们分析了在线性回归任务中，对角线线性网络的损失曲面与训练动态，其中网络参数在训练过程中受到各向同性正态噪声的扰动。这种噪声可被解释为随机形式的尖锐感知最小化（SAM），我们证明了其作用与损失的尖锐度密切相关。具体而言，噪声在参数上诱导出分数阶范数惩罚的加权混合，促使各层以较快速率实现平衡，并改变底层损失曲面，使其倾向于由真实参数经收缩-阈值算子得到的解。我们表明，层间平衡等价于在所有相同线性预测器的分解中最小化平均尖锐度及海森矩阵的迹。此外，我们精确刻画了正态扰动的噪声水平作为正则化参数的作用，包括其对收缩、阈值和平衡速度的影响。
原文摘要 · Abstract (English)
We analyze the landscape and training dynamics of diagonal linear networks in a linear regression task, with the network parameters being perturbed by isotropic normal noise during training. The addition of such noise may be interpreted as a stochastic form of sharpness-aware minimization (SAM) and we prove several results that relate its action on the underlying landscape and training dynamics to the sharpness of the loss. In particular, the noise induces a weighted mixture of fractional norm penalties on the network parameters, which forces the individual layers to balance at a fast rate and changes the underlying landscape to favor solutions that result from a shrinkage-thresholding operator applied to the true parameter. We show that balancing the layers equates to minimizing the average sharpness, as well as the trace of the Hessian matrix, among all possible factorizations of the same linear predictor. Further, we characterize how the noise level of the normal perturbations acts as a regularization parameter, with exact descriptions of its effect on shrinkage, thresholding, and balancing speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。