揭示了上下文学习本质是贝叶斯推断，解释了大模型如何快速适应新任务。
In-Context Learning Is Provably Bayesian Inference: A Generalization Theory for Meta-Learning
- 从元学习视角分解上下文学习风险，分离出贝叶斯差距与后验方差。
- 证明小样本下模型能快速收敛到最优任务算法，仅需少量示例即可消除任务混合带来的不确定性。
- 为Transformer的上下文学习提供理论支撑，适合研究大模型机制与泛化能力者阅读。
本文构建了上下文学习（ICL）的有限样本统计理论，基于可容纳多种任务类型的元学习框架。提出一种严谨的风险分解方法，将总风险拆分为两个正交分量：贝叶斯差距与后验方差。贝叶斯差距衡量训练模型对贝叶斯最优上下文预测器的逼近程度；对于均匀注意力Transformer，我们推导出该差距的非渐近上界，明确揭示其依赖于预训练提示数量及其上下文长度。后验方差是模型无关的风险，代表任务本身的内在不确定性。关键发现是：该分量仅由真实任务难度决定，而任务混合引起的不确定性随少量上下文示例呈指数级衰减。上述结果统一了对ICL的理解：Transformer在预训练中选择最优元算法，并在测试时迅速收敛至真实任务的最优算法。
原文摘要 · Abstract (English)
This paper develops a finite-sample statistical theory for in-context learning (ICL), analyzed within a meta-learning framework that accommodates mixtures of diverse task types. We introduce a principled risk decomposition that separates the total ICL risk into two orthogonal components: Bayes Gap and Posterior Variance. The Bayes Gap quantifies how well the trained model approximates the Bayes-optimal in-context predictor. For a uniform-attention Transformer, we derive a non-asymptotic upper bound on this gap, which explicitly clarifies the dependence on the number of pretraining prompts and their context length. The Posterior Variance is a model-independent risk representing the intrinsic task uncertainty. Our key finding is that this term is determined solely by the difficulty of the true underlying task, while the uncertainty arising from the task mixture vanishes exponentially fast with only a few in-context examples. Together, these results provide a unified view of ICL: the Transformer selects the optimal meta-algorithm during pretraining and rapidly converges to the optimal algorithm for the true task at test time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。