用直觉逻辑重新定义下一个词预测,构建新型神经网络架构
Modeling Next-Token Prediction as Left-Nested Intuitionistic Implication
- 将词序建模为左嵌套蕴含链,非交换组合保持顺序
- 通过构造性证明扩展实现预测,等价于乘法RNN结构
- 适合研究逻辑与神经网络融合的学者,理论性强
我们提出 extit{箭头语言模型}(Arrow Language Model),基于直觉逻辑对下一个词预测进行建模。不同于注意力机制中加性嵌入混合的方式,该模型将前缀编码为左嵌套蕴含链,其结构通过非交换复合保持顺序。下一个词预测对应于 extit{假言推理}(modus ponens),序列处理则成为在柯里-霍华德对应下的构造性证明延伸。基于Prolog的专用定理证明器验证了神经模型的基本性质,包括交换与非交换序列、单词与多词预测选择之间的关系。我们表明,从下一词预测的证明论解释出发,自然导出乘法RNN结构,并提出了一个实用的低秩神经实现,将该模型置于Transformer与状态空间模型之间进行定位。
原文摘要 · Abstract (English)
We introduce the \emph{Arrow Language Model}, a neural architecture derived from an intuitionistic-logic interpretation of next-token prediction. Instead of representing tokens as additive embeddings mixed by attention, we encode a prefix as a \emph{left-nested implication chain} whose structure preserves order through non-commutative composition. Next-token prediction corresponds to \emph{modus ponens}, and sequence processing becomes constructive proof extension under the Curry--Howard correspondence. Our Prolog-based specialized theorem provers validate fundamental properties of the neural models, among which relations between commutative vs. non-commutative sequencing and single-token vs. multi-token prediction choices. We show that a neural architecture equivalent to multiplicative RNNs arises naturally from a proof-theoretic interpretation of next-token prediction as nested intuitionistic implication, we present a practical low-rank neural realization and position the model relative to Transformers and state-space models. Keywords: logic-based derivation of neural architectures, intuitionistic implicational logic, token-as-operator neural models, state-space models, alternatives to transformer-based foundational models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。