提出一种稳定且可扩展的分数差估计方法，提升小样本下生成模型迁移性能。
Sobolev Regularized Score Difference Estimation in Diffusion Models
- 基于Sobolev正则化构建分数差估计器，保证统计一致性
- 在低样本情况下显著提升稳定性，理论收敛率达O(n^(-(s-1)/(d+2s-2)))
- 适用于心电图生成等真实任务，迁移效果优于非正则化方法
估计两个Stein分数函数的差异是生成建模中的基础问题。尤其在迁移学习中，分数差为将预训练模型适配到新目标分布提供了机制；在基于扩散模型的后训练方法（如判别器引导）中也自然出现。现有分数差估计器或缺乏统计一致性，或在高维下难以扩展。本文提出一种基于Sobolev正则化的统计一致且可扩展的分数差估计器，在小样本情形下有效保证一致性并稳定训练。理论上，我们建立了均方误差下的收敛率O(n^(-(s-1)/(d+2s-2)))，其中d为维度，s为底层密度的光滑性；并给出最小最大下界˜Ω(n^(-2(s-1)/(d+2s)))。实验表明，该方法在小样本下稳定性显著优于现有方法。我们在真实任务中验证其有效性，包括心电图信号生成的迁移学习，下游分类性能明显优于非正则化估计器。
原文摘要 · Abstract (English)
Estimating the difference of two Stein's score functions is a fundamental problem in generative modeling. In particular, score differences arise naturally in transfer learning, where the score difference provides the mechanism for adapting a pre-trained model to a new target distribution, and in diffusion model-based post-training methods such as discriminator guidance. Existing estimators for score differences in these settings either lack of statistical consistency or are difficult to scale up in high-dimensions. We propose a statistically consistent and scalable estimator for score differences based on Sobolev regularization, which plays a crucial role in ensuring consistency and stablizing the training in the small-sample regime. Mathematically, we establish a convergence rate of $O(n^{-\frac{s-1}{d+2s-2}})$ where $d$ is the dimension and $s$ denotes the smoothness of the underlying densities, and provide a minimax lower bound of $\tildeΩ(n^{-\frac{2(s-1)}{d+2s}})$ (in mean-squared error). Empirically, our estimator exhibits significantly improved stability in small-sample regimes compared to existing methods. We demonstrate its effectiveness on real-world tasks, including transfer learning for ECG signal generation, where it substantially outperforms non-regularized score difference estimators in downstream classification performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。