证明了满足特定条件的Transformer可实现精确贝叶斯推断
Transformer Architectures as Complete Bayes Processes: A Formal Proof in the Measure-Theoretic Kernel Framework
- 在测度论核框架下构建抽象层级,逐层证明贝叶斯联合分布条件成立时更新核等于后验
- 推导出块级显式贝叶斯公式并证明其归一化,揭示注意力机制生成合法概率分布
- 为Transformer提供形式化贝叶斯解释,适合关注模型可信性与理论基础的研究者
我们提出一个完整的形式化证明:当Transformer内部更新机制满足贝叶斯联合分布条件时,其能实现精确的贝叶斯后验推断。基于测度论核框架,定义从核心贝叶斯Transformer到带显式更新核的语义Transformer、含QKV/注意力/残差/MLP流水线的完整块结构,再到多层堆叠的抽象层级,并在每一层证明贝叶斯联合语义蕴含更新核几乎处处等于后验。针对块级结构,通过Radon-Nikodym微分导出显式贝叶斯公式并证明其归一化。此外,证明softmax注意力机制在键上诱导出有效概率分布,建立抽象核框架与具体注意力实现之间的桥梁。该框架不依赖额外架构假设,仅基于马尔可夫核结构,明确指出了Transformer块严格贝叶斯成立的充要条件。本质上,当此联合分布条件满足时,Transformer前向计算在形式上等价于严谨的贝叶斯后验更新。
原文摘要 · Abstract (English)
We present a complete formal proof that transformer architectures, when their internal update mechanisms satisfy a Bayes joint-distribution condition, implement exact Bayesian posterior inference. Working within the measure-theoretic kernel framework, we define a hierarchy of abstractions -- from the core Bayesian transformer, through semantic transformers with explicit update kernels, to full transformer blocks with QKV/attention/residual/MLP pipelines, and finally multilayer stacks -- and prove at each level that the Bayes joint semantics implies the update kernel equals the posterior almost everywhere. For the block-level architecture, we derive the explicit Bayes formula through Radon-Nikodym differentiation and prove its normalization. We additionally prove that the softmax attention mechanism induces a valid probability distribution over keys, establishing the bridge between the abstract kernel framework and concrete attention implementations. The framework makes no architectural assumptions beyond the Markov kernel structure and exposes explicit conditions under which a transformer block is provably Bayesian. In essence, when this joint distribution condition is satisfied, the forward computation of a Transformer is formally equivalent to a rigorous Bayesian posterior update.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。