用逻辑与自动机理论揭示Transformer的能力边界
The Role of Logic and Automata in Understanding Transformers
- 通过逻辑与自动机理论分析Transformer的计算能力
- 揭示其在处理形式语言时的强弱表现
- 适合关注模型理论基础的研究者
Transformer的出现近年来催生了强大的大型语言模型（LLMs）。尽管如此，我们对Transformer能力的理解仍十分有限。本文回顾了过去几年在回答「Transformer能做什么」这一问题上的快速进展。特别地，我们将看到逻辑、自动机（以及电路复杂性的一些帮助）在解答该问题中所起的关键作用。同时，文章还提及了逻辑、自动机、验证与Transformer交叉领域中的若干开放问题。
原文摘要 · Abstract (English)
The advent of transformers has in recent years led to powerful and revolutionary Large Language Models (LLMs). Despite this, our understanding on the capability of transformers is still meager. In this invited contribution, we recount the rapid progress in the last few years to the question of what transformers can do. In particular, we will see the integral role of logic and automata (also with some help from circuit complexity) in answering this question. We also mention several open problems at the intersection of logic, automata, verification and transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。