单次训练时，最大似然等同于贝叶斯，无需担心过拟合。
Why you don't overfit, and don't need Bayes if you only train for one epoch
- 单次遍历数据时，最大似然直接优化真实数据分布损失
- 贝叶斯平均在相同条件下也优化同一目标函数
- 适用于大模型单轮训练场景，无需复杂贝叶斯推断
在数据充足且每样本仅训练一次（或仅训练一个周期）的设定下，标准的最大似然训练会优化真实数据生成过程（DGP）损失，该损失等价于测试损失。我们进一步证明，贝叶斯模型平均同样优化这一目标，只是通过考虑有限数据带来的不确定性期望。由于最大似然在单周期设置下与贝叶斯推断优化相同的客观目标，因此在这些情况下，我们不期待贝叶斯方法能带来过拟合抑制或校准性能上的优势。这解释了为何在大型语言模型等常以单轮或极少数轮次训练的领域中，贝叶斯方法的重要性逐渐降低。
原文摘要 · Abstract (English)
Here, we show that in the data-rich setting where you only train on each datapoint once (or equivalently, you only train for one epoch), standard "maximum likelihood" training optimizes the true data generating process (DGP) loss, which is equivalent to the test loss. Further, we show that the Bayesian model average optimizes the same objective, albeit while taking the expectation over uncertainty induced by finite data. As standard maximum likelihood training in the single-epoch setting optimizes the same objective as Bayesian inference, we argue that we do not expect Bayesian inference to offer any advantages in terms of overfitting or calibration in these settings. This explains the diminishing importance of Bayes in areas such as LLMs, which are often trained with one (or very few) epochs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。