PFN模型内部存在可提取的谱结构,能实现快速贝叶斯推断。
Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks
- 通过探针实验发现注意力分数中线性可解码出主导谱轴。
- 谱方向比随机方向有效十倍以上,且在真实时间序列上仍成立。
- 提出滤波器银行解码器,可显式重建可移植的贝叶斯核函数。
先验-数据拟合网络(PFNs)可在单次前向传播中实现近似贝叶斯推断,但其内部表征仍不透明。我们提供机制性证据表明,PFNs学习到结构化的谱表示,并可提取为显式核函数。在三种架构(包括公开的TabPFN)上,谱信息可从潜在注意力分数中线性解码,沿主导主轴组织。激活补丁与目标子空间干预表明,该信息因果地用于预测,且集中于低维子空间,谱方向的效果比随机方向高一个数量级。关键的是,这些特性在TabPFN上对合成分布外输入及真实世界时间序列(航空乘客、牛奶产量)均成立,说明这是连续回归任务中基于PFN风格近似推断的涌现特征,而非训练先验的产物。其次,我们引入滤波器银行解码器,将冻结的PFN潜在表示映射为显式谱密度,基于Bochner定理重构平稳核函数。所得核函数支持与迭代基线相当的高斯过程回归,仅需一次前向传播,证明PFN先验不仅是隐式的,还可显式恢复为可移植的贝叶斯对象。
原文摘要 · Abstract (English)
Prior-Data Fitted Networks (PFNs) enable amortized Bayesian inference in a single forward pass, yet their internal representations remain opaque. It is unknown whether PFNs encode identifiable Bayesian structure or merely memorize input-output mappings. We provide mechanistic evidence that PFNs learn structured spectral representations and that these can be extracted as explicit kernels. First, probing experiments across three architectures, including the publicly released TabPFN, show that spectral information is linearly decodable from the latent attention score and organized along a dominant principal axis. Activation patching and targeted subspace interventions establish that this information is causally used for prediction and concentrated in a low-dimensional subspace, with spectral directions an order of magnitude more effective than random ones. Crucially, these properties hold on TabPFN with both synthetic out-of-distribution inputs and real-world time series (Airline Passengers, Milk Production), indicating they are emergent features of PFN-style amortization over continuous regression tasks rather than artifacts of training prior. Second, we introduce a Filter Bank Decoder that maps frozen PFN latents to explicit spectral densities, reconstructing stationary kernels via Bochner's theorem. The resulting kernels support GP regression competitive with iterative baselines while requiring only a single forward pass, demonstrating that PFN priors are not merely implicit but are explicitly recoverable as portable Bayesian objects.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。