改进了神经网络优化中的粒子混沌传播，提升模型集成效果。
Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble
- 通过改进对数Sobolev不等式，优化粒子系统逼近性能。
- 消除正则化系数带来的指数依赖，降低优化复杂度。
- 提出有理论保障的模型集成方法，适合深度学习训练场景。
均场Langevin动力学（MFLD）是从两层神经网络的噪声梯度下降在均场极限下导出的优化方法。近期，其混沌传播（PoC）特性受到关注，因其能定量刻画粒子数量与迭代次数对优化复杂度的影响。Chen等（2022）证明，有限粒子引起的近似误差在时间上保持均匀，并随粒子数增加而减小。本文在神经网络训练设定下，进一步改进该工作的缺陷对数Sobolev不等式，建立了改进的PoC结果，消除了优化复杂度中粒子近似项对正则化系数的指数依赖。作为应用，我们提出一种基于PoC的模型集成策略，具备理论保证。
原文摘要 · Abstract (English)
Mean-field Langevin dynamics (MFLD) is an optimization method derived by taking the mean-field limit of noisy gradient descent for two-layer neural networks in the mean-field regime. Recently, the propagation of chaos (PoC) for MFLD has gained attention as it provides a quantitative characterization of the optimization complexity in terms of the number of particles and iterations. A remarkable progress by Chen et al. (2022) showed that the approximation error due to finite particles remains uniform in time and diminishes as the number of particles increases. In this paper, by refining the defective log-Sobolev inequality -- a key result from that earlier work -- under the neural network training setting, we establish an improved PoC result for MFLD, which removes the exponential dependence on the regularization coefficient from the particle approximation term of the optimization complexity. As an application, we propose a PoC-based model ensemble strategy with theoretical guarantees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。