提出一种闭式校正方法，显著降低生成模型训练中的采样偏差。
Analytical Correction for Subsampling Bias in Drifting Models
- 基于批内统计量解析估算并校正软最大值归一化带来的偏差
- 理论证明偏差从1/n降至1/n²，且不增加方差
- 代码仅增两行，小批量下效果尤其明显，适合生成模型优化
漂移模型是一类能够一步生成的生成模型，其训练依赖于数据分布与当前生成器分布的吸引力与排斥力加权中心点组合。实际中仅能获取每个分布的一个大小为n的迷你批次样本，每个中心点由经验估计近似。本文首先证明，迷你批次中心点通常是对目标中心点的有偏估计，其点对点偏差为O(1/n)，源于软最大值自归一化。修正该偏差需对全分布取期望，但不可行。为此，我们通过批内统计量近似主偏差项，提出解析偏差校正（ABC），一种闭式插值调整方法。理论上，ABC将偏差从O(1/n)降至O(1/n²)，不引入一阶方差增长，并保持修正后中心点在凸包内。实践中，ABC只需额外两行代码，编译执行下开销可忽略。小型实验验证了理论上的1/n与1/n²缩放关系。在CIFAR-10上，ABC降低FID分数且训练更快，尤其在小批量n时收益最大。
原文摘要 · Abstract (English)
Drifting models are capable one-step generative models trained to follow a drifting field. The field combines attractive and repulsive softmax-weighted centroids over the data and current-generator distributions. In practice, only a minibatch of $n$ samples from each distribution is available, and each centroid is approximated by an empirical estimate. In this paper, we begin by showing that the minibatch centroid is in general a biased estimator of the target centroid, with a pointwise $O(1/n)$ bias arising from softmax self-normalization. Correcting this bias requires the expectation over the full distribution, which is intractable. We instead approximate the leading bias term from in-batch statistics and propose Analytical Bias Correction (ABC), a closed-form plug-in adjustment. We prove that ABC reduces the bias from $O(1/n)$ to $O(1/n^2)$, introduces no first-order increase in total variance, and preserves convex-hull containment of the corrected centroid. In practice, ABC requires only two additional lines of code and has negligible wall-time overhead under compiled execution. Toy experiments confirm the theoretical $O(1/n)$ and $O(1/n^2)$ scaling. On CIFAR-10, ABC reduces FID and trains faster, with the largest gains at small $n$, where the bias is most significant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。