大学习率能防止扩散模型过拟合，因它让神经网络无法精确拟合训练数据。
Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
- 用大学习率训练时，神经网络无法收敛到过拟合解。
- 在小噪声下，最优得分函数极不规则，难以精确拟合。
- 实验表明该机制适用于多维数据，适合研究生成模型泛化问题者。
去噪得分匹配在基于扩散的生成模型中至关重要。然而，其经验最优得分会导致过拟合，即生成样本复制训练数据。实践中即便无显式正则化，过拟合程度也较轻。本文揭示了大学习率带来的隐式正则化机制：在小噪声条件下，经验最优得分具有高度不规则性；当使用足够大的学习率进行随机梯度下降时，神经网络无法稳定收敛至任意小过量风险的局部极小值。因此，学到的得分无法无限接近经验最优得分，从而缓解过拟合。分析基于一维数据与两层神经网络，实验验证了学习率在防止过拟合中的关键作用，且该现象在高维设置中依然成立。
原文摘要 · Abstract (English)
Denoising score matching plays a pivotal role in the performance of diffusion-based generative models. However, the empirical optimal score--the exact solution to the denoising score matching--leads to memorization, where generated samples replicate the training data. Yet, in practice, only a moderate degree of memorization is observed, even without explicit regularization. In this paper, we investigate this phenomenon by uncovering an implicit regularization mechanism driven by large learning rates. Specifically, we show that in the small-noise regime, the empirical optimal score exhibits high irregularity. We then prove that, when trained by stochastic gradient descent with a large enough learning rate, neural networks cannot stably converge to a local minimum with arbitrarily small excess risk. Consequently, the learned score cannot be arbitrarily close to the empirical optimal score, thereby mitigating memorization. To make the analysis tractable, we consider one-dimensional data and two-layer neural networks. Experiments validate the crucial role of the learning rate in preventing memorization, even beyond the one-dimensional setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。