揭示熵正则化中不对称性对模型学习的影响
Asymmetry of the Relative Entropy in the Regularization of Empirical Risk Minimization
- 对比两种熵正则化方式，分析其差异与机制
- 发现两者均使模型支持集坍缩至参考分布支持集
- 指出该现象会削弱训练数据的证据作用，适合理论研究者
在经验风险最小化（ERM）的相对熵正则化（ERM-RER）框架下，分析了相对熵不对称性的影响。考虑两种正则化形式：(a) 待优化测度相对于参考测度的相对熵（类型I ERM-RER）；(b) 参考测度相对于待优化测度的相对熵（类型II ERM-RER）。主要结果是刻画了类型II ERM-RER问题的解及其关键性质。通过与广为人知的类型I进行比较，凸显了熵不对称性的效应。分析表明，在两种情况下，相对熵正则化都会迫使解的支持集坍缩到参考测度的支持集内，引入强归纳偏置，从而抵消训练数据提供的证据。最后证明，类型II正则化等价于对经验风险函数进行适当变换后的类型I正则化。
原文摘要 · Abstract (English)
The effect of relative entropy asymmetry is analyzed in the context of empirical risk minimization (ERM) with relative entropy regularization (ERM-RER). Two regularizations are considered: $(a)$ the relative entropy of the measure to be optimized with respect to a reference measure (Type-I ERM-RER); and $(b)$ the relative entropy of the reference measure with respect to the measure to be optimized (Type-II ERM-RER). The main result is the characterization of the solution to the Type-II ERM-RER problem and its key properties. By comparing the well-understood Type-I ERM-RER with Type-II ERM-RER, the effects of entropy asymmetry are highlighted. The analysis shows that in both cases, regularization by relative entropy forces the solution's support to collapse into the support of the reference measure, introducing a strong inductive bias that negates the evidence provided by the training data. Finally, it is shown that Type-II regularization is equivalent to Type-I regularization with an appropriate transformation of the empirical risk function.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。