Transformer可基于上下文学习贝叶斯网络的序列生成
Transformers Simulate MLE for Sequence Generation in Bayesian Networks
- 用上下文MLE让Transformer估计贝叶斯网络条件概率
- 训练后能按学习到的概率自回归生成新序列
- 为大模型作为概率生成器提供理论支持
Transformer在自然语言处理等序列任务中取得显著成功，但其理论能力仍不清晰。本文研究Transformer在贝叶斯网络中基于上下文最大似然估计（MLE）进行自回归序列生成的能力。设定中，上下文由一组独立生成的、符合贝叶斯网络的数据序列构成。我们证明：存在一种简单Transformer模型，既能根据上下文估计贝叶斯网络的条件概率，又能据此自回归生成符合该网络的新样本。通过大量实验，我们进一步验证该模型不仅理论上存在，还能通过训练有效获得。分析揭示了Transformer学习复杂概率模型的潜力，有助于理解大型语言模型作为强大序列生成器的本质。
原文摘要 · Abstract (English)
Transformers have achieved significant success in various fields, notably excelling in tasks involving sequential data like natural language processing. Despite these achievements, the theoretical understanding of transformers' capabilities remains limited. In this paper, we investigate the theoretical capabilities of transformers to autoregressively generate sequences in Bayesian networks based on in-context maximum likelihood estimation (MLE). Specifically, we consider a setting where a context is formed by a set of independent sequences generated according to a Bayesian network. We demonstrate that there exists a simple transformer model that can (i) estimate the conditional probabilities of the Bayesian network according to the context, and (ii) autoregressively generate a new sample according to the Bayesian network with estimated conditional probabilities. We further demonstrate in extensive experiments that such a transformer does not only exist in theory, but can also be effectively obtained through training. Our analysis highlights the potential of transformers to learn complex probabilistic models and contributes to a better understanding of large language models as a powerful class of sequence generators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。