arXiv:2608.03921cs.AIcs.NE2026-08

Transformer推理时动态生成参数,实现按提示定制的思维转换。

The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections

  • 用输入动态生成变换参数,取代静态权重,形成可变处理流程。
  • 长提示下动态处理贡献超过静态处理,体现强提示敏感性。
  • 为模型可解释性与类脑语言机制提供新视角,适合关注模型本质的研究者。

本文重新诠释了Transformer在推理阶段的表现。针对大语言模型仅复制训练中统计规律的“随机鹦鹉”观点,我们提出:Transformer通过推理过程构建并应用依赖于提示的动态变换,其参数由输入生成。这种处理方式称为序列级交互式动态并行处理(SIDPP)。Transformer被视为一种以概念变换概念的系统:标记向量是待变换的概念,由矩阵和向量定义的参数化变换则是变换概念。这些变换可为静态(训练固定)或动态(从输入生成)。在结构上,它们对应一组简单神经网络。Transformer的创新在于输出-权重互连机制,即部分网络的输出决定其他网络的权重,同时保留常规的输出-输入连接。通过此类互连,系统根据提示构造变换,并用于修改标记表示。动态处理的贡献随提示长度增长,在某些情况下甚至超过静态处理,这一现象称为强提示敏感性。该解释对可解释性、预测性、可控性以及小型可持续系统的构建具有意义。此外,由于人脑神经系统具备实现SIDPP所需的机制,我们推测人类语言处理本身可能就是一种由功能结构与Transformer相似的系统所实现的SIDPP形式。

原文摘要 · Abstract (English)

This paper offers a new interpretation of the Transformer during inference. Against the "stochastic parrot" view that large language models merely reproduce statistical regularities learned in training, we argue that Transformers construct and apply prompt-dependent transformations whose parameters are generated during inference. We call this form of processing SIDPP: Sequence-level Interactive Dynamic Parallel Processing. The Transformer is interpreted as a system that transforms concepts by means of concepts. Token vectors are the concepts to be transformed; parameterized transformations defined by matrices and vectors are the transforming concepts. These may be static, when fixed through training, or dynamic, when generated from the input sequence. Mechanically, they correspond to groups of simple neural networks. The Transformer's architectural novelty lies in output-weight interconnections, through which the outputs of some networks determine the weights of others, alongside ordinary output-input interconnections. By means of these interconnections, the system constructs transformations from the prompt and uses them to modify token representations. The contribution of dynamic processing grows with prompt length and may equal or exceed that of static processing, a phenomenon we call strong prompt sensitivity. This account bears on interpretability, predictability, control, and the design of smaller, more sustainable systems. Finally, because the human neural system possesses mechanisms similar to those required for SIDPP, we argue that a form of SIDPP may, in principle, be realized in the cerebral cortex. We therefore conjecture that human language processing may itself be a form of SIDPP produced by a functional architecture relevantly similar to that of the Transformer.

Transformer动态推理可解释性类脑计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。