深度高斯过程的极限行为存在关键带宽阈值,低于阈值时可生成非退化、非高斯的复杂分布。
How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs

- 通过分析每层为向量高斯过程的深度结构,发现带宽存在临界值
- 低于临界值时先验收敛到非退化非高斯分布,且坐标间有依赖
- 实验证实阈值随维度变化,揭示深层模型潜在复杂特性
组合先验描述了深度贝叶斯模型中分层函数的通用性质,其中权重随机的深度神经网络是典型例子。在宽网络极限下,先验是一个具有深度相关核的高斯过程,其随深度增长的行为已通过该核得到广泛研究。本文研究另一种情形:每一层本身是向量值高斯过程,目标同样是理解先验随深度增长的极限行为。已有高斯过程研究指出,对于RBF核及特定带宽范围 $r$,先验在极限下会退化至常数函数集合——这在概率模型中无用。本文建立若干新结果:首先,我们识别出一个尖锐的带宽阈值 $r_c(d) = Θ( ext{√}d)$,超过此阈值则极限退化,强化了先前界限;其次,更重要的是,当 $r$ 低于阈值 $r_c(d)$ 时,先验收敛至极限分布 $π_{\bar{Z}}$。我们还证明这些分布是非退化的、非高斯的,且坐标间存在不消失的依赖关系。与先前退化情形相反,深度高斯过程先验确实可具备非平凡极限。实验上,我们在多维 $d$ 下验证了该阈值,并展示了极限分布 $π_{\bar{Z}}$ 的复杂多峰行为——这一区域随 $d$ 增大而日益狭窄,若不知阈值则极难识别。
原文摘要 · Abstract (English)
Compositional priors describe the generic properties of layered functions in deep Bayesian models, where deep neural networks with random weights are a canonical example.In the wide-network limit, the prior is a Gaussian process with a depth-dependent kernel, and its behaviour as depth grows has been extensively studied through this kernel. Here, we study another case, where each layer itself is a vector valued Gaussian process, and our aim is similarly to understand the limiting behaviour of the prior as depth grows. Previous GP work has established that for the RBF kernel and a certain range of bandwidths $r$, the prior degenerates in the limit, converging to the set of constant functions -- which is not useful as a probabilistic model. In this paper we establish several new results. First, we identify a sharp bandwidth threshold $r_c(d) = Θ(\sqrt{d})$ above which the limit is degenerate, strengthening the earlier bounds. Second, and more importantly, we show that for $r$ below the threshold $r_c(d)$ the prior converges to a limit distribution $π_{\bar{Z}}$. We also prove that these distributions are non-degenerate and non-Gaussian, with non-vanishing dependence between coordinates. In contrast to the previously known degenerate regime, deep Gaussian process priors can therefore admit non-trivial limits. Empirically, we verify the threshold across a range of dimensions $d$, and demonstrate a complex multimodal behaviour of the limit distributions $π_{\bar{Z}}$ -- a regime that becomes increasingly narrow with $d$ and would be hard to identify without knowing the threshold.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。