Transformer可实现完整贝叶斯推断，无需训练即可学习复杂模型后验。
Can Transformers Learn Full Bayesian Inference in Context?
- 基于拟合网络与连续归一化流，构建上下文贝叶斯推断通用框架。
- 在真实数据集上，后验采样质量接近先进MCMC或变分推断方法。
- 适合对上下文学习、贝叶斯推理感兴趣的科研人员参考。
Transformer已成为深度学习主流架构，具备广泛的应用和出色的上下文学习（ICL）能力。尽管机制尚未完全明晰，但ICL已展现出独特价值：可在不进行额外训练的情况下实现上下文学习。本文进一步深化对ICL的理解，证明Transformer可在上下文中执行常见统计模型的完整贝叶斯推断。具体而言，我们提出一个通用框架，融合先前拟合网络与连续归一化流的思想，能够对广义线性模型和潜因子模型等复杂模型推导后验分布。在真实数据集上的大量实验表明，该ICL方法生成的后验样本质量与不依赖上下文的前沿MCMC或变分推断方法相当。相关代码已开源：https://github.com/ArikReuter/ICL_for_Full_Bayesian_Inference。
原文摘要 · Abstract (English)
Transformers have emerged as the dominant architecture in the field of deep learning, with a broad range of applications and remarkable in-context learning (ICL) capabilities. While not yet fully understood, ICL has already proved to be an intriguing phenomenon, allowing transformers to learn in context -- without requiring further training. In this paper, we further advance the understanding of ICL by demonstrating that transformers can perform full Bayesian inference for commonly used statistical models in context. More specifically, we introduce a general framework that builds on ideas from prior fitted networks and continuous normalizing flows and enables us to infer complex posterior distributions for models such as generalized linear models and latent factor models. Extensive experiments on real-world datasets demonstrate that our ICL approach yields posterior samples that are similar in quality to state-of-the-art MCMC or variational inference methods that do not operate in context. The source code for this paper is available at https://github.com/ArikReuter/ICL_for_Full_Bayesian_Inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。