arXiv:2602.04596stat.MLcs.LG2026-02被引 6

分解贝叶斯Transformer的不确定性,区分认知与随机误差。

Uncertainty Decomposition for Bayes-Filtered Transformers via Bayesian Predictive Inference

  • 基于贝叶斯预测推断,建立预测中心极限定理
  • 实现认知不确定性随上下文增长而减小,且在稀疏区域最高
  • 适用于表格预测模型TabPFN,结果符合预期分布

贝叶斯滤波Transformer通过在先验预测分布上元学习序列,以近似后验预测分布。它们能在单次前向传播中输出总预测不确定性,但不显式表示后验分布,因此无法使用标准方法分离认知与随机不确定性。本文从贝叶斯预测推断(BPI)视角解决此问题。主要成果是在监督设置下,于贝叶斯预测推断文献中条件最弱的情形下,建立了预测中心极限定理(CLT)。该定理表明,在给定观测上下文时,极限预测分布的后验渐近服从高斯分布;其方差量化了认知不确定性。我们将该框架应用于TabPFN——一种用于表格预测的先进基础模型。所得可信区间在上下文长度增加时接近名义频率覆盖率,且分解结果基本符合预期:认知不确定性随上下文长度增加而减小,在上下文数据覆盖范围内的稀疏区域最高,而随机不确定性在类别重叠的决策边界附近占主导。

原文摘要 · Abstract (English)

Bayes-filtered transformers are transformers meta-learned on sequences from a prior predictive distribution to approximate the corresponding posterior predictive distribution. They output total predictive uncertainty in a single forward pass but never explicitly represent a posterior distribution, making the standard route to separating aleatoric from epistemic uncertainty unavailable. We address this challenge through the lens of Bayesian predictive inference (BPI). Our main result is a predictive Central Limit Theorem (CLT) for supervised settings under conditions that are among the weakest known in the BPI literature. The CLT characterises the posterior of the limiting predictive distribution given an observed context as asymptotically Gaussian; the variance of this Gaussian quantifies epistemic uncertainty. We apply the framework to TabPFN, a Bayes-filtered transformer that is a state-of-the-art foundation model for tabular prediction. The resulting credible bands achieve near-nominal frequentist coverage as context length grows, and the decomposition largely matches standard desiderata: epistemic uncertainty shrinks with context length and is highest in sparsely observed regions within the span of the context data, while aleatoric uncertainty dominates near decision boundaries where classes overlap.

贝叶斯推理不确定性量化Transformer表格预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。