通过几何隐式偏置先验,更精准地评估过参数模型的泛化性能。
PAC--Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors
- 在商空间上进行贝叶斯分析,消除参数对称性带来的干扰
- 实验显示先验使均值商空间KL下降40.69%,证书收紧21.40%
- 适合研究模型隐式正则化与泛化理论的学者
过参数化模型常具有连续参数对称性,不同参数对应相同预测器。我们表明,PAC-贝叶斯分析应作用于商预测空间:将先验与后验映射到商空间,可保持经验与总体吉布斯风险,同时消除仅由同一预测器的不同参数化差异导致的非负KL项。单纯商化无法决定使用何种先验。我们为每个预测器构造一个标准参数化形式,并考虑其等价参数化的几何体积。这将中性参考先验转化为数据无关的隐式偏置先验,逼近理想但不可行的后验匹配先验(依赖训练数据以最小化KL项)。所得证明证书更紧,当此几何诱导先验与学习到的商后验的KL小于中性先验时成立。我们在傅里叶回归(哈达玛参数化)和查询-键注意力中测试该预测,使用普通SGD且无显式正则化。傅里叶-哈达玛实验中,隐式偏置先验使平均商空间KL降低40.69%,平均PAC-贝叶斯证书降低21.40%;查询-键注意力中改进较小,确认了该效应的条件依赖性。
原文摘要 · Abstract (English)
Overparameterized models often have continuous parameter symmetries, so different parameters define the same predictor. We show that PAC--Bayesian analysis should be performed on the quotient predictor space: pushing a prior and posterior to the quotient preserves the empirical and population Gibbs risks while removing the nonnegative KL contribution caused solely by how the two distributions differ among parameterizations of the same predictor. Quotienting alone does not determine which prior to use. We construct a canonical choice of one parameterization for each predictor and account for the geometric volume of its equivalent parameterizations. This transforms a neutral reference prior into a data-independent prior that reflects the model's implicit bias. It approximates the ideal but inadmissible posterior-matched prior, which would minimize the KL term by depending on the training data. The resulting certificate is tighter exactly when this geometry-induced prior has smaller KL divergence from the learned quotient posterior than the neutral prior. We test this prediction in Fourier regression with a Hadamard parameterization and in Query-Key attention, using ordinary SGD without an explicit regularizer. The implicit-bias prior reduces the mean quotient-space KL by \(40.69\%\) and the mean PAC--Bayes certificate by \(21.40\%\) in the Fourier-Hadamard experiment. The smaller, prior-scale-dependent improvement in Query-Key attention confirms the predicted conditional nature of the effect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。