先验信息决定记忆好坏，过拟合何时有害有明确边界
Is Memorization Helpful or Harmful? Prior Information Sets the Threshold
- 基于贝叶斯框架分析过参数线性模型中的先验影响
- 当噪声达到特定阈值时，记忆或避免过拟合成为最优选择
- 适用于理解模型泛化能力的理论边界，适合理论研究者
我们研究任意估计方法在过参数化线性模型下的训练误差与泛化误差关系，采用贝叶斯设定并考虑一般先验。发现先验分布 π 决定了关键因素，明确给出了最优泛化需满足的条件：(i) 训练误差接近噪声水平（即记忆是必需的），或 (ii) 接近噪声级别（即过拟合有害）。这些现象发生在噪声达到由先验的费舍尔信息和方差参数决定的阈值时。
原文摘要 · Abstract (English)
We examine the connection between training error and generalization error for arbitrary estimating procedures, working in an overparameterized linear model under general priors in a Bayesian setup. We find determining factors inherent to the prior distribution $π$, giving explicit conditions under which optimal generalization necessitates that the training error be (i) near interpolating relative to the noise size (i.e., memorization is necessary), or (ii) close to the noise level (i.e., overfitting is harmful). Remarkably, these phenomena occur when the noise reaches thresholds determined by the Fisher information and the variance parameters of the prior $π$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。