单次种子评估在贝叶斯深度学习中可能误导,需关注方差轨迹。
A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning

- 通过50次重复实验,发现不同方法的CRPS方差变化不平滑,存在可复现的波动峰。
- 在首尔自行车数据集上,单次种子估计误差可达93.6%,±10%覆盖概率低至5.9%。
- 建议报告方差轨迹摘要,重点评估高方差区域,避免依赖单一结果。
在小样本场景下,评估指标(如连续排名概率评分,CRPS)的单次种子均值本身是随机变量,却常被当作方法的稳定属性报告。我们研究了该做法失效的情形。基于六个回归数据集的50次独立重复实验,发现不同方法的CRPS方差轨迹差异显著,且不总是符合平滑的幂律衰减。使用学习异方差头的方法(如MAP和深度集成)在真实数据集上会于中等训练规模处产生明显且可复现的方差峰值,而MC Dropout和贝叶斯反向传播通常表现为平稳的方差收缩。这些峰值具有实际影响:在首尔自行车数据集的方差峰值处,单次种子的MAP估计相对RMSE高达93.6%,落入±10%重复运行均值范围的概率降至5.9%。我们证明局部CRPS方差能直接反映单次种子估计误差,所有真实数据集的斯皮尔曼相关系数均超过0.96。幂律拟合质量与单调性共同提供方法轨迹规律性的紧凑总结。将标准异方差目标替换为β-NLL可显著降低不规则行为,表明异方差训练目标可能加剧不稳定性。实践者应同时报告轨迹摘要,并集中重复评估在高方差区域。
原文摘要 · Abstract (English)
In limited-data settings, a single endpoint mean of an evaluation metric such as the Continuous Ranked Probability Score (CRPS) is itself a random variable, yet it is routinely reported as if it were a stable property of the method. We study when this practice fails. Using 50 independent repetitions across six regression datasets, we show that CRPS variance trajectories differ substantially across methods and are not always well described by a smooth power-law decay. Methods with a learned heteroscedastic variance head, namely MAP and Deep Ensembles, can develop pronounced, reproducible variance peaks at intermediate training sizes on real datasets, whereas MC Dropout and Bayes by Backprop typically show smooth variance contraction. These peaks have direct practical consequences: at the variance peak on Seoul Bike, the relative RMSE of a single-seed MAP estimate reaches 93.6\%, and the probability of falling within \(\pm 10\%\) of the repeated-run mean drops to 5.9\%. We show that local CRPS variance provides a direct signal of single-seed estimation error, with Spearman correlations above 0.96 on every real dataset. Power-law fit quality and monotonicity together provide compact method-level summaries of trajectory regularity. Finally, replacing the standard heteroscedastic objective with \(β\)-NLL substantially reduces the irregular behavior, consistent with the view that the heteroscedastic training objective contributes to the instability. Practitioners should report trajectory summaries alongside endpoint means and concentrate repeated evaluation in high-variance regions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。