用热力学响应理论统一解释奇异贝叶斯模型的复杂性与预测波动。
Thermodynamic Response Functions in Singular Bayesian Models
- 通过后验退火构建温度依赖的响应函数体系
- 揭示了WAIC、WBIC与奇异波动的统一热力学关系
- 适合研究模型复杂性与结构重排的理论学习者
奇异统计模型(包括混合模型、矩阵分解和神经网络)因参数不可识别和费雪信息几何退化而违背常规渐近性质。尽管奇异学习理论通过实对数规范阈值(RLCT)和奇异波动等不变量描述边缘似然行为,但这些量难于实际操作解读。同时,广泛使用的WAIC和WBIC与底层奇异几何看似脱节。本文表明,后验退火诱导后验分布的一参数变形,其相关可观测量生成一系列热力学响应函数。一个普遍协方差恒等式将温化期望的导数与后验波动联系起来,使WAIC、WBIC和奇异波动纳入统一响应框架。在此框架中,经典奇异学习理论量获得自然热力学解释:RLCT决定自由能主导斜率,奇异波动对应温化自由能曲率,WAIC度量预测波动。我们形式化了一种可观测代数,以消除不可识别方向,从而在奇异模型中构造出结构上有意义的序参量。在典型奇异例子(对称高斯混合、降秩回归、过参数化神经网络)中,我们实证展示了退火下的相变样行为:序参量坍缩,敏感度峰值,复杂度度量与后验几何重组一致。结果表明,热力学响应理论为奇异贝叶斯学习中的复杂性、预测可变性和结构重组提供了自然组织框架。
原文摘要 · Abstract (English)
Singular statistical models-including mixtures, matrix factorization, and neural networks-violate regular asymptotics due to parameter non-identifiability and degenerate Fisher geometry. Although singular learning theory characterizes marginal likelihood behavior through invariants such as the real log canonical threshold and singular fluctuation, these quantities remain difficult to interpret operationally. At the same time, widely used criteria such as WAIC and WBIC appear disconnected from underlying singular geometry. We show that posterior tempering induces a one-parameter deformation of the posterior distribution whose associated observables generate a hierarchy of thermodynamic response functions. A universal covariance identity links derivatives of tempered expectations to posterior fluctuations, placing WAIC, WBIC, and singular fluctuation within a unified response framework. Within this framework, classical quantities from singular learning theory acquire natural thermodynamic interpretations: RLCT governs the leading free-energy slope, singular fluctuation corresponds to curvature of the tempered free energy, and WAIC measures predictive fluctuation. We formalize an observable algebra that quotients out non-identifiable directions, allowing structurally meaningful order parameters to be constructed in singular models. Across canonical singular examples-including symmetric Gaussian mixtures, reduced-rank regression, and overparameterized neural networks-we empirically demonstrate phase-transition-like behavior under tempering. Order parameters collapse, susceptibilities peak, and complexity measures align with structural reorganization in posterior geometry. Our results suggest that thermodynamic response theory provides a natural organizing framework for interpreting complexity, predictive variability, and structural reorganization in singular Bayesian learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。