SGD像在分形地形上扩散，可视为改进的贝叶斯采样器。
Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent
- 将SGD视为分形地形上的扩散过程，用贝叶斯框架解释其行为
- 实验验证权重扩散特性与分形结构一致
- 为理解SGD与贝叶斯采样的关系提供新视角
我们发现随机梯度下降（SGD）的行为与贝叶斯统计相关，因为其本质是在分形损失景观上的扩散过程，而分形维度可通过纯贝叶斯方式建模。由此表明，SGD可被看作一种考虑分形结构带来的可及性约束的修正贝叶斯采样器。通过分析训练中权重的扩散行为，我们对这一结果进行了实验验证。这些发现揭示了决定学习过程的关键因素，并似乎解答了SGD与纯贝叶斯采样之间的关联问题。
原文摘要 · Abstract (English)
We show that the behavior of stochastic gradient descent is related to Bayesian statistics by showing that SGD is effectively diffusion on a fractal landscape, where the fractal dimension can be accounted for in a purely Bayesian way. By doing this we show that SGD can be regarded as a modified Bayesian sampler which accounts for accessibility constraints induced by the fractal structure of the loss landscape. We verify our results experimentally by examining the diffusion of weights during training. These results offer insight into the factors which determine the learning process, and seemingly answer the question of how SGD and purely Bayesian sampling are related.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。