提出贝叶斯充分表示的新定义,明确最优预测所需最少信息。
Bayes-Sufficient Representations in Supervised Learning

- 用贝叶斯动作规则定义信息相关性,与损失函数紧密绑定。
- 在二分类中,零一损失对应类别,平方损失对应条件均值。
- 实验证明冗余信息可被去除,最小表示能实现最优预测。
表示学习常被理解为保留对预测有用的信息。本文针对固定监督问题,定义了贝叶斯充分表示:若某表示能让预测头实现贝叶斯最优决策,则称其充分。该定义使相关信息依赖于损失函数。在几乎必然唯一的贝叶斯动作情况下,关键对象是贝叶斯商,它识别出需采取相同最优行动的输入。当表示细化该商时即为充分,信息等价于商时为贝叶斯最小。该框架自然关联到属性诱导:零一损失需贝叶斯类别,平方损失需条件均值,二元预测中布雷尔损失需条件概率,对数损失或严格可分评分规则则需预测分布。通过控制的有限实验、学习的神经瓶颈实验及真实数据iNaturalist分类精炼实验,展示了充分性、最小性与非必要信息保留之间的区别。对于固定监督问题,分布与损失决定贝叶斯动作,贝叶斯动作决定商,商决定实现贝叶斯最优预测所需的最小信息。
原文摘要 · Abstract (English)
Representation learning is often described as preserving the information in an input that is relevant for prediction. This work asks what relevance means for a fixed supervised decision problem. A representation is defined to be Bayes-sufficient for a joint distribution and loss if some prediction head can use it to implement a Bayes-optimal action rule. This makes the target information loss-dependent. In the almost-surely unique Bayes-action case, the relevant object is a Bayes quotient, which identifies inputs that require the same Bayes-optimal action. A representation is sufficient when it refines this quotient, and Bayes-minimal when it is informationally equivalent to it. The framework connects naturally to property elicitation: zero-one loss requires the Bayes class, squared loss the conditional mean, Brier loss the conditional probability in binary prediction, and log loss or strictly proper scoring rules the predictive distribution. Controlled finite experiments, learned neural bottleneck experiments, and a real-data iNaturalist taxonomic refinement experiment illustrate the distinction between sufficiency, minimality, and retained non-required information. For a fixed supervised problem, the distribution and the loss determine the Bayes action, the Bayes action determines the quotient, and the quotient determines the minimal information required for Bayes-optimal prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。