用自注意力网络实现逻辑程序的上下推推导，揭示大模型隐含的推理能力。
Implementing Derivations of Definite Logic Programs with Self-Attention Networks
- 通过层级自注意力+前馈网络构建逻辑推导机制。
- 成功实现一类逻辑公式的上下文推导，证明结构可行性。
- 为理解大模型的隐式逻辑推理提供新视角，适合研究模型可解释性者。
本文提出，受限的逻辑推导可由自注意力网络实现。目标是验证基于Transformer的大语言模型具备逻辑推理能力。我们通过分析自注意力网络——Transformer的核心组件——揭示其潜在逻辑推导能力。方法不依赖自然语言语义，而是基于逻辑推导操作。研究显示，带有前馈网络（FFNs）的层次化自注意力结构可实现一类逻辑公式的上行推导；同样结构也可实现下行推导。结果表明，大语言模型可能隐含逻辑推理能力。
原文摘要 · Abstract (English)
In this paper we propose that a restricted version of logical inference can be implemented with self-attention networks. We are aiming at showing that LLMs (Large Language Models) constructed with transformer networks can make logical inferences. We would reveal the potential of LLMs by analyzing self-attention networks, which are main components of transformer networks. Our approach is not based on semantics of natural languages but operations of logical inference. %point of view. We show that hierarchical constructions of self-attention networks with feed forward networks (FFNs) can implement top-down derivations for a class of logical formulae. We also show bottom-up derivations are also implemented for the same class. We believe that our results show that LLMs implicitly have the power of logical inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。