重新定义模型泛化复杂度,从参数空间转向预测行为本身
PAC-Bayes Beyond Parameter Space: Behavioral Equivalence, Z-Information, and Exact Complexity Decomposition
- 用可测行为映射分解模型配置空间,分离行为选择与具体实现
- 首次给出经典PAC-Bayes复杂度的精确分解:行为选择项+实现细节项
- 提出Z信息量,量化行为等价配置带来的隐藏复杂度
PAC-Bayes理论通过控制后验与先验在假设表示空间上的KL散度来提供泛化保证。然而,预测风险仅依赖于假设所引发的预测行为,而非其内部具体实现。在过参数化系统中,多种不同配置可能产生相同预测行为,但传统PAC-Bayes的KL散度无法区分行为不确定性与行为等价实现之间的差异。本文通过可测行为映射,利用测度分解将配置空间的概率测度分解为行为分布与纤维上条件分布之积,实现了对经典PAC-Bayes KL散度的精确分解:由行为选择项和纤维内条件KL期望组成。定义Z信息为该实现级贡献的负值,即KL散度与仅关于行为不确定性的复杂度之差。进一步证明行为选择项具有精确变分表征——最小化所有生成相同行为分布的后验中的KL散度,由规范的纤维对称代表达成。最后指出对称性、保持行为的方向、纤维几何及纤维保持扰动下的不变性均源于同一行为映射结构。这些结果确立了预测行为是PAC-Bayes复杂度的自然对象。
原文摘要 · Abstract (English)
PAC-Bayes theory provides generalization guarantees by controlling the Kullback--Leibler (KL) divergence between posterior and prior distributions over a chosen hypothesis representation. However, predictive risk depends only on the predictive behavior induced by a hypothesis, not on the particular internal realization that implements that behavior. In over-parameterized systems, many distinct configurations induce identical predictive behavior, yet the classical PAC-Bayes KL divergence does not distinguish uncertainty over predictive behavior from variation among behaviorally equivalent realizations. We show that this distinction induces an exact structural decomposition of classical PAC-Bayes complexity. We formalize behavioral equivalence through a measurable behavior map and use measure disintegration to decompose probability measures on the configuration space into a distribution over predictive behaviors and conditional distributions over behavioral fibers. This yields an exact decomposition of the classical PAC-Bayes KL divergence into a behavior-selection term and a realization-level term given by an expected conditional KL within fibers. We define Z-information as the negative of this realization-level contribution: the exact gap between the KL divergence and the complexity of uncertainty over predictive behavior alone. We further show that the behavior-selection term admits an exact variational characterization: it is the minimum KL divergence among all posteriors inducing the same distribution over predictive behaviors, attained by a canonical fiber-symmetrized representative. Finally, we show that symmetry, behavior-preserving directions, fiber geometry, and invariance under fiber-preserving perturbations arise naturally from the same behavior-map structure. Together, these results identify predictive behavior as the natural object of PAC-Bayes complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。