解析神经过程近似高斯过程的三大代价，给出模型设计改进方案。
Three Costs of Amortizing Gaussian Process Inference with Neural Processes
- 分解近似误差为标签污染、信息瓶颈和摊销误差三部分。
- 信息瓶颈项随维度指数衰减，标签污染项基本恒定仅噪声项随样本数下降。
- 建议用位置预测方差、二阶池化替代均值聚合以提升精度。
神经过程通过学习从上下文集到预测分布的 $O(n)$ 映射，取代了高斯过程 $O(n^3)$ 的精确后验。针对一类潜在神经过程，本文界定了其与精确高斯过程预测之间的 Kullback--Leibler (KL) 散度，将其分解为三个可解释来源：标签污染（神经过程使用标签值估计本应与标签无关的量）、信息瓶颈（有限维表示无法刻画完整上下文几何结构），以及共享编码器带来的摊销误差。信息瓶颈项在平方指数核下以 $O(e^{-cd^{2/d_x}})$ 衰减，在 Matérn-$ν$ 核下以 $O(d^{-2ν/d_x})$ 衰减，直接关联架构尺寸与核平滑性及输入维度。标签污染项整体为 $O(1)$，仅观测噪声部分以 $O(1/n)$ 下降，揭示了通过标签依赖表示传递不确定性估计的持久成本。结果为该类模型的架构设计提供了依据：建议在高斯过程近似框架中仅根据上下文位置预测方差，并用二阶池化替代均值聚合以缩小主导摊销差距。
原文摘要 · Abstract (English)
Neural processes amortize Gaussian process inference, replacing the exact $O(n^3)$ posterior with a learned $O(n)$ map from context sets to predictive distributions. For a class of latent neural processes, we bound the Kullback--Leibler (KL) divergence between the GP and LNP predictives, decomposing it into three interpretable sources, namely label contamination as the neural process uses label values to estimate a quantity that is label-independent in the exact GP, an information bottleneck because the finite-dimensional representation cannot resolve the full context geometry, and amortization error from a single encoder network shared across all contexts. The bottleneck truncation term decays in the representation dimension $d$ as $O(e^{-cd^{2/d_x}})$ for squared-exponential kernels on $\mathbb{R}^{d_x}$ where $c > 0$ is a kernel-dependent constant and as $O(d^{-2ν/d_x})$ for Matérn-$ν$ kernels, directly linking architecture sizing to kernel smoothness and input dimension. The label contamination term is $O(1)$ in general, with only the observation-noise component decaying as $O(1/n)$, identifying a persistent cost of routing uncertainty estimation through a label-dependent representation. These results characterize the costs of amortization within the analyzed class and yield architectural recommendations to predict variance from context locations alone in the GP-amortization regime, and replace mean aggregation with second-order pooling to close the dominant amortization gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。