arXiv:2504.10845cs.CLcs.AI2025-04

将Transformer看作上下文敏感语言生成器,解释其为何能产生类人智能输出。

Moving Beyond Next-Token Prediction: Transformers are Context-Sensitive Language Generators

  • 把Transformer拆解为上下文窗口、注意力和自回归三部分,突破传统理解
  • 认为逐词预测本质是动态近似左文敏感语言规则,解释类人输出来源
  • 为生成式AI提供理论基础,适合研究模型机制的学者参考

大型语言模型(LLMs)由Transformer驱动,展现出类人智能能力,但其内在机制仍不清晰。本文提出新框架,将LLMs视为概率性左文敏感语言(CSLs)生成器。我们假设Transformer可有效分解为三个基本组件:上下文窗口、注意力机制与自回归生成框架。该分解使计算模型更灵活且可解释,突破了注意力与自回归不可分割的传统观点。我们认为,下一词预测可被理解为对左文敏感语言生成规则的概率性动态近似,为简单词预测如何生成类人智能输出提供了直观解释。由于所有CSL均为左文敏感(Penttonen, 1974),我们得出结论:Transformer在随机意义上逼近CSL,而后者被广泛认为是人类智能的模型。这一解释弥合了形式语言理论与Transformer生成能力之间的鸿沟,为生成式AI的理论与应用发展奠定基础。本研究视角将深化对大模型的理解及其未来潜力。

原文摘要 · Abstract (English)

Large Language Models (LLMs), powered by Transformers, have demonstrated human-like intelligence capabilities, yet their underlying mechanisms remain poorly understood. This paper presents a novel framework for interpreting LLMs as probabilistic left context-sensitive languages (CSLs) generators. We hypothesize that Transformers can be effectively decomposed into three fundamental components: context windows, attention mechanisms, and autoregressive generation frameworks. This decomposition allows for the development of more flexible and interpretable computational models, moving beyond the traditional view of attention and autoregression as inseparable processes. We argue that next-token predictions can be understood as probabilistic, dynamic approximations of left CSL production rules, providing an intuitive explanation for how simple token predictions can yield human-like intelligence outputs. Given that all CSLs are left context-sensitive (Penttonen, 1974), we conclude that Transformers stochastically approximate CSLs, which are widely recognized as models of human-like intelligence. This interpretation bridges the gap between Formal Language Theory and the observed generative power of Transformers, laying a foundation for future advancements in generative AI theory and applications. Our novel perspective on Transformer architectures will foster a deeper understanding of LLMs and their future potentials.

Transformer语言模型形式语言生成机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。