分析了Transformer在符号推理中的最大推理步数极限。

Limit Analysis for Symbolic Multi-step Reasoning Tasks with Information Propagation Rules Based on Transformers

• 基于注意力传播规则构建符号推理模型框架。
• 理论证明单次前向传播下推理步数上限为3^{L-1}，下限为2^{L-1}。
• 适合研究大模型推理能力边界与注意力机制的学者参考。

Transformer 能够执行推理任务，但其内在机制仍不明确。本文提出一套基于 Transformer 的信息传播规则，并利用符号推理任务对推理步骤的极限进行理论分析。结果表明，在单次前向传播中，具有 L 层注意力层的模型，其推理步数的极限范围为 $O(3^{L-1})$ 到 $O(2^{L-1})$。该分析揭示了模型深度与推理能力之间的理论边界，为理解 Transformer 的推理潜力提供了形式化依据。

原文摘要 · Abstract (English)

Transformers are able to perform reasoning tasks, however the intrinsic mechanism remains widely open. In this paper we propose a set of information propagation rules based on Transformers and utilize symbolic reasoning tasks to theoretically analyze the limit reasoning steps. We show that the limit number of reasoning steps is between $O(3^{L-1})$ and $O(2^{L-1})$ for a model with $L$ attention layers in a single-pass.

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。