扩散模型虽对混合权重不敏感,仍能准确恢复其值。
Diffusion models recover accurate mixture weights despite score function insensitivity

- 通过分析噪声过程中的得分敏感性,揭示权重恢复机制。
- 在任意维度高斯混合分布中,权重误差与DSM损失同阶。
- 适合研究生成模型对分布参数估计能力的学者。
基于得分的生成模型表现出一个令人困惑的现象:尽管通常覆盖目标多模态分布的所有模式,却可能无法学习到正确的模式相对幅度(即混合权重)。本文通过将扩散得分匹配(DSM)损失与生成样本中混合权重估计误差关联起来,解决了这一矛盾。我们证明,即使目标得分对混合权重不敏感,只要中间噪声水平下的得分信息能反映权重变化,生成样本仍可准确恢复权重。为此提出扩散得分敏感性指数(DSSI),量化DSM损失对参数变化的响应程度。理论表明,在任意维度高斯混合分布下,权重估计误差与DSM损失同阶。实验显示,在典型噪声调度下,基准数据分布的敏感性随去噪过程逐渐显现,且敏感性数值可预测模型对权重的恢复能力。此外,噪声调度的选择会影响扩散敏感性,进而导致模式放大。该框架不仅适用于混合权重,还可推广至目标分布中任何定性参数的恢复。
原文摘要 · Abstract (English)
Score-based generative models exhibit a puzzling behavior: they often appear to cover all modes of a target multimodal distribution and yet may fail to learn the correct relative mode amplitudes, which can be interpreted as mixture weights. We resolve this apparent paradox by relating the diffusion score matching (DSM) loss to the error in estimating mixture weights from generated samples. We show that, even when the target score is insensitive to mixture weights, generated samples can recover the weights accurately if the scores at intermediate noise levels are informative about the weights. Accordingly, we define the diffusion score sensitivity index (DSSI) as the variation in the DSM loss relative to changes in a parameter. We then show that the DSSI governs the accuracy with which the parameter of the target distribution can be estimated from generated samples. For Gaussian mixtures in arbitrary dimensions, we prove that the mixture weight estimation errors are on the same order as the DSM loss under mild conditions. Empirically, we show the emergence of sensitivity during the noising process of benchmark data distributions under typical noise schedules, and that these sensitivity values predict how well a well-trained model recovers mixture weights. Furthermore, we show that the choice of noise schedule can reduce diffusion sensitivity, leading to mode amplification. Although we focus on mixture weights, the proposed sensitivity framework governs the recovery of any qualitative parameter of the target distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。