变分学习让神经网络找到更平坦的解，提升稳定性。
Variational Learning Finds Flatter Solutions at the Edge of Stability
- 通过控制后验分布形状和采样数量，增强隐式正则化。
- 在ResNet、ViT等模型上验证，理论与实证高度一致。
- 适合关注模型泛化与训练稳定性的研究者。
变分学习（VL）近年来在训练深度神经网络中表现出色。尽管其成功可部分由PAC-Bayes界、最小描述长度和边缘似然等理论解释，但其隐式正则化机制仍不清晰。本文基于边缘稳定性（EoS）框架分析了VL的隐式正则化。已有研究表明梯度下降能寻找平坦解，我们进一步证明VL可找到更平坦的解。该结论通过调控变分后验的形状及训练时的后验采样数量实现。推导过程沿用标准EoS文献方法：先针对二次问题得出结果，再扩展至深度神经网络。我们在多种大型网络（如ResNet和ViT）上进行实证验证，发现理论预测与实际结果高度吻合。本工作是首个分析变分学习在边缘稳定性下动态行为的研究。
原文摘要 · Abstract (English)
Variational Learning (VL) has recently gained popularity for training deep neural networks. Part of its empirical success can be explained by theories such as PAC-Bayes bounds, minimum description length and marginal likelihood, but little has been done to unravel the implicit regularization in play. Here, we analyze the implicit regularization of VL through the Edge of Stability (EoS) framework. EoS has previously been used to show that gradient descent can find flat solutions and we extend this result to show that VL can find even flatter solutions. This result is obtained by controlling the shape of the variational posterior as well as the number of posterior samples used during training. The derivation follows in a similar fashion as in the standard EoS literature for deep learning, by first deriving a result for a quadratic problem and then extending it to deep neural networks. We empirically validate these findings on a wide variety of large networks, such as ResNet and ViT, to find that the theoretical results closely match the empirical ones. Ours is the first work to analyze the EoS dynamics of VL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。