将注意力机制解释为基于上下文的贝叶斯推断，揭示深度与残差的统计作用。
Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics
- 单次注意力计算等价于上下文经验分布下的核加权后验均值。
- 深度通过粒子动力学优化分布，长距离连接作为查询实现后验推断。
- 无需显式噪声调度即可实现有效去噪，适合理解注意力的统计原理。
我们在全标记噪声条件下研究仅含注意力的最小化Transformer模型，发现其具有两阶段经验贝叶斯解释。单次注意力计算是在上下文定义的经验分布上进行核加权后验均值估计。深度通过粒子动力学（第一阶段）精炼该分布，而长程跳跃连接将噪声输入作为查询用于后验推断（第二阶段），揭示了深度与注意力残差的不同统计角色。该框架在最小设置中揭示了上下文本身诱导的深度依赖能量景观，主导上下文内推理。我们证明，固定核带宽与有限积分时域即可实现有效去噪，无需显式噪声调度，从而建立深度与噪声间的合理关系。进一步地，对于一类良好行为的先验，我们建立了后验均值恢复保证：经验估计量在渐近条件下收敛至贝叶斯最优预测器。通过与反向扩散极限的关联，我们的结果为注意力提供了一种无显式密度建模的统计解释——即基于样本的上下文后验估计。
原文摘要 · Abstract (English)
We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a depth-dependent energy landscape governing in-context inference. We show that effective denoising can emerge without an explicit noise schedule: a fixed kernel bandwidth and finite integration horizon suffice, yielding a principled depth-noise relationship. We further establish a posterior-mean recovery guarantee for a class of well-behaved priors, where the empirical estimator converges to the Bayes-optimal predictor under asymptotic conditions. Connecting these dynamics to reverse-diffusion limits, our results provide a statistical interpretation of attention as in-context inference via sample-based posterior estimation, without explicit density modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。