FID评估结果受随机种子影响极大,同一模型多次生成结果差异显著。
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation

- 将FID视为训练与采样双随机变量,系统测量其波动性
- 重训练模型导致的FID变化是重采样的3.2倍,主因是初始化和数据顺序
- 建议用多种子评估并报告误差范围,1.3%内差异不具说服力
Frechet Inception Distance (FID) 是图像生成质量的默认评估标准,但多数论文仅报告单次训练、单个采样种子下的单一数值。若重新训练模型或仅重采样,该数值是否可重现?本文将FID视为训练与生成种子双重变量下的随机变量,在数百个基于类条件ImageNet 256x256的SiT网络上直接测量其方差。发现:(a) 使用相同训练流程但不同种子重训练模型,所导致的FID变化(在Inception特征空间中)是固定模型下重采样的3.2倍;(b) 此差异由三个因素驱动:随机初始化、数据顺序及流匹配损失中的每步高斯噪声;(c) 增加计算量或模型规模几乎不缩小方差,FID变异系数(CoV)稳定在1%-2%区间;(d) 采用逐单元无分类器引导调优可使方差减半,但改变最优种子分布,幸运的训练种子可比不幸者节省最多2倍计算量。据此,建议新评估协议:在逐单元最优引导下评估,将实测约1.3%的CoV作为阈值,任何低于此的差异视为不显著,并报告多个训练种子的误差棒而非单一数值。
原文摘要 · Abstract (English)
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance directly on several hundred SiT networks trained on class-conditional ImageNet 256x256. We report surprising findings: (a) Retraining the model using the same recipe with a different seed moves FID 3.2x more (in Inception feature space) than redrawing samples from a fixed network. (b) That gap is driven by three factors: random initialisation, data ordering, and the per-step Gaussian noise of the flow-matching loss. (c) Increasing compute or model size barely tightens the spread, holding the FID coefficient of variation (CoV) inside a 1-2% band. (d) Per-cell classifier-free-guidance tuning halves the spread but reshuffles which seeds work best, and a lucky training seed reaches the same FID with up to 2x less compute than an unlucky one. Based on these findings, we recommend a new FID evaluation protocol: evaluate under per-cell optimal guidance, treat any FID gap below the empirically measured ~1.3% CoV as inconclusive, and report an error bar over several training seeds rather than a single FID number.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。