揭示去噪得分匹配的方差不稳定性,提出改进方法提升训练稳健性。
Heteroscedasticity of Denoising Score Matching with Generalised Smooth Noise
- 发现去噪得分匹配存在异方差问题,方差随噪声水平和数据几何变化
- 推导理想加权函数使方差均一化,显著降低训练梯度波动
- 给出现有启发式权重的理论解释,适用于扩散模型与高阶得分估计
得分匹配(Score Matching, SM)是一种无需计算归一化常数即可估计分布对数密度梯度的强大框架,广泛应用于统计估计、能量模型及现代基于扩散的生成模型。实践中,这些模型几乎全依赖可计算的去噪得分匹配(Denoising Score Matching, DSM)作为替代。这一普遍性引发一个根本问题:DSM 是否真能实现“免费得分匹配”?本文证明,DSM 并非完美替代品——其目标函数本质上具有异方差性,模型参数的方差会随噪声水平和底层数据几何结构而不可预测地波动。这种不稳定性根植于 DSM 的数学结构。为此,我们推导出一个理想的加权函数,使方差均一化,得到 DSM 的同方差推广。由于理想权重通常无法直接获取,我们通过泰勒展开提出一种实用近似权重,虽牺牲部分统计最优性,但有效降低训练过程中的梯度方差。该结果为各向同性高斯扩散中已有启发式权重提供了理论依据。我们在多种扰动分布及高阶得分场景下验证了理论的有效性。
原文摘要 · Abstract (English)
Score Matching (SM) is a powerful framework for estimating the log-density derivatives of a distribution without calculating its normalizing constants. This capability has made it a cornerstone across multiple domains, from classical sta- tistical estimation and energy-based models to modern diffusion-based generative models. In practice, these models rely almost exclusively on Denoising Score Matching (DSM) as a tractable proxy for score matching. This ubiquity naturally raises a fundamental question: Is DSM truly "score matching for free"? In this work, we demonstrate that DSM is not a perfect substitute. We prove that the denoising objective is inherently heteroscedastic, the variance of model parame- ters fluctuates unpredictably based on both noise levels and the underlying data geometry. This instability is baked into the mathematical structure of the DSM. To address this, we derive an ideal weighting function that equalizes this variance, yielding a homoscedastic generalization of DSM. Since the ideal weights are of- ten empirically inaccessible, we show that a practical approximation weighting function via Taylor expansion reduces gradient variance during training, at the cost of statistical optimality. Notably, this provides a theoretical justification for an existing heuristic weight used in Isotropic Gaussian Diffusion. We validate our theory across different perturbed distributions and for higher-order scores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。