arXiv:2502.09998stat.MLcs.LG2025-02

提出新方法用经验损失估算学习系数,更准更稳。

Estimation of the Learning Coefficient Using Empirical Loss

  • 基于瓦塔纳贝引入的'经验损失'构造新估算方法
  • 实验显示偏差和方差均低于已有方法
  • 适合关注模型泛化评估的理论与应用研究者

学习系数在分析信息准则(如WAIC、WBIC)性能中起关键作用,由渡边澄夫提出,用于评估模型泛化能力。在常规统计模型中,学习系数为 d/2,d 为参数空间维数;一般情形下,它等于由Kullback-Leibler散度与先验分布导出的zeta函数极点阶数的绝对值。然而,除降秩回归等特例外,学习系数无法解析求解。渡边提出数值估计方法,今井进一步改进以提升收敛性。这些方法利用WBIC的渐近行为,已被证明随样本量增大具统计一致性。本文提出一种根本不同的新数值估计方法,基于瓦塔纳贝引入的“经验损失”这一新量。数值实验表明,本方法相比渡边与今井方法具有更低偏差与更低方差。我们还提供了理论分析解释其优越性,并给出实证支持。

原文摘要 · Abstract (English)

The learning coefficient plays a crucial role in analyzing the performance of information criteria, such as the Widely Applicable Information Criterion (WAIC) and the Widely Applicable Bayesian Information Criterion (WBIC), which Sumio Watanabe developed to assess model generalization ability. In regular statistical models, the learning coefficient is given by d/2, where d is the dimension of the parameter space. More generally, it is defined as the absolute value of the pole order of a zeta function derived from the Kullback-Leibler divergence and the prior distribution. However, except for specific cases such as reduced-rank regression, the learning coefficient cannot be derived in a closed form. Watanabe proposed a numerical method to estimate the learning coefficient, which Imai further refined to enhance its convergence properties. These methods utilize the asymptotic behavior of WBIC and have been shown to be statistically consistent as the sample size grows. In this paper, we propose a novel numerical estimation method that fundamentally differs from previous approaches and leverages a new quantity, "Empirical Loss," which was introduced by Watanabe. Through numerical experiments, we demonstrate that our proposed method exhibits both lower bias and lower variance compared to those of Watanabe and Imai. Additionally, we provide a theoretical analysis that elucidates why our method outperforms existing techniques and present empirical evidence that supports our findings.

学习系数信息准则模型评估数值方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。