提出新型块采样MAC-Bayes泛化界,更紧致且适用性更强。
Block-Sample MAC-Bayes Generalization Bounds
- 基于训练数据块设计新泛化界,仅依赖子集信息
- 数值示例中原PAC-Bayes界失效,新界仍有效
- 首次证明高概率版本难以实现,理论意义显著
我们提出一类新型的块采样MAC-Bayes泛化界(均值近似正确)。与通常以高概率成立的PAC-Bayes界不同,MAC-Bayes界针对期望泛化误差进行约束。所提界可视为已有PAC-Bayes界期望形式的推广。相比标准PAC-Bayes界,新界中的散度项仅依赖于训练数据的子集(或称“块”)。该方法有望显著提升传统PAC-Bayes与MAC-Bayes界的紧致性。通过简单数值例子说明:无论先验如何选择,原PAC-Bayes界均为真空;而所提界在合适块大小下保持有限。此外,我们探讨了此类MAC-Bayes界是否存在高概率版本(即类似形式的PAC-Bayes界),并以反例否定:一般情况下无法建立一个当新界以$\ ext{O}(n^{-1/2})$速率趋于零时,仍以快于$\ ext{O}(1/\log n)$速率趋零、且误差概率对数依赖的PAC-Bayes界。
原文摘要 · Abstract (English)
We present a family of novel block-sample MAC-Bayes bounds (mean approximately correct). While PAC-Bayes bounds (probably approximately correct) typically give bounds for the generalization error that hold with high probability, MAC-Bayes bounds have a similar form but bound the expected generalization error instead. The family of bounds we propose can be understood as a generalization of an expectation version of known PAC-Bayes bounds. Compared to standard PAC-Bayes bounds, the new bounds contain divergence terms that only depend on subsets (or \emph{blocks}) of the training data. The proposed MAC-Bayes bounds hold the promise of significantly improving upon the tightness of traditional PAC-Bayes and MAC-Bayes bounds. This is illustrated with a simple numerical example in which the original PAC-Bayes bound is vacuous regardless of the choice of prior, while the proposed family of bounds are finite for appropriate choices of the block size. We also explore the question whether high-probability versions of our MAC-Bayes bounds (i.e., PAC-Bayes bounds of a similar form) are possible. We answer this question in the negative with an example that shows that in general, it is not possible to establish a PAC-Bayes bound which (a) vanishes with a rate faster than $\mathcal{O}(1/\log n)$ whenever the proposed MAC-Bayes bound vanishes with rate $\mathcal{O}(n^{-1/2})$ and (b) exhibits a logarithmic dependence on the permitted error probability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。