通过核平滑降低扩散模型得分噪声,缓解过拟合问题。
Kernel-Smoothed Scores for Denoising Diffusion: A Bias-Variance Study
- 用核平滑处理经验得分,降低反向扩散中的方差
- 理论证明可将得分正则化等效为增大训练数据量
- 适合关注生成模型泛化能力的研究者
扩散模型在高保真生成任务中表现优异,但可能因过拟合有限数据集而产生记忆现象,导致反向时间SDE采样结果几乎全是训练样本。本文将经验得分视为真实得分的带噪版本,发现其协方差矩阵渐近于加权数据主成分分析(PCA)。在高维情形下,小时间极限使噪声方差急剧放大,同时空间相关性减弱。为此引入核平滑的经验得分,并分析其偏差-方差权衡。推导出修正后反向SDE生成分布与真实分布之间KL散度的渐近界。得分正则化效果等同于增加训练集规模,有助于防止记忆现象。前向扩散的谱分解表明,在真实数据分布满足一定光滑性条件下可实现更优的方差控制。核平滑后的反向扩散可重述为指向对数指数双核密度估计器(LED-KDE)的梯度下降过程,揭示了两种正则化机制:初始高斯核在全空间各向同性扩散质量,次级核在得分空间沿数据流形集中并扩散质量。因此,仅通过简单正则化即可有效缓解记忆现象、提升泛化性能。数值实验在合成数据和MNIST上验证了上述结论。
原文摘要 · Abstract (English)
Diffusion models now set the benchmark in high-fidelity generative sampling, yet they can, in principle, be prone to memorization. In this case, their learned score overfits the finite dataset so that the reverse-time SDE samples are mostly training points. In this paper, we interpret the empirical score as a noisy version of the true score and show that its covariance matrix is asymptotically a re-weighted data PCA. In large dimension, the small time limit makes the noise variance blow up while simultaneously reducing spatial correlation. To reduce this variance, we introduce a kernel-smoothed empirical score and analyze its bias-variance trade-off. We derive asymptotic bounds on the Kullback-Leibler divergence between the true distribution and the one generated by the modified reverse SDE. Regularization on the score has the same effect as increasing the size of the training dataset, and thus helps prevent memorization. A spectral decomposition of the forward diffusion suggests better variance control under some regularity conditions of the true data distribution. Reverse diffusion with kernel-smoothed empirical score can be reformulated as a gradient descent drifted toward a Log-Exponential Double-Kernel Density Estimator (LED-KDE). This perspective highlights two regularization mechanisms taking place in denoising diffusions: an initial Gaussian kernel first diffuses mass isotropically in the ambient space, while a second kernel applied in score space concentrates and spreads that mass along the data manifold. Hence, even a straightforward regularization-without any learning-already mitigates memorization and enhances generalization. Numerically, we illustrate our results with several experiments on synthetic and MNIST datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。