研究对称性如何影响优化，发现噪声投影可带来新形式隐式正则化。
On the implicit regularization of Langevin dynamics with projected noise
- 在群作用正交方向上投影噪声，构造新型随机优化过程。
- 当初始与目标分布均保持对称时，等价于带额外漂移的各向同性扩散。
- 该漂移项由群轨道的平均曲率决定，适用于理解过参数模型优化机制。
我们研究了在等距群作用正交方向上投影噪声的Langevin动力学。该数学模型旨在揭示对称性对过参数化模型中随机梯度下降的影响。主要结果表明：当初始和目标密度均在群作用下不变时，带有投影噪声的Langevin动力学在分布上等价于带有额外漂移项的各向同性扩散，该漂移项与群轨道的负对数体积成正比。通过在群自身上构造一个中间过程实现两过程的耦合，并将该额外漂移识别为轨道的平均曲率。
原文摘要 · Abstract (English)
We study Langevin dynamics with noise projected onto the directions orthogonal to an isometric group action. This mathematical model is introduced to shed new light on the effects of symmetry on stochastic gradient descent for over-parametrized models. Our main result identifies a novel form of implicit regularization: when the initial and target density are both invariant under the group action, Langevin dynamics with projected noise is equivalent in law to Langevin dynamics with isotropic diffusion but with an additional drift term proportional to the negative log volume of the group orbit. We prove this result by constructing a coupling of the two processes via a third process on the group itself, and identify the additional drift as the mean curvature of the orbits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。