arXiv:2606.01532cs.LGcs.CC2026-06

无需位置编码,滑动窗口变压器仍可实现通用计算。

Rethinking the Role of Positional Encoding: Sliding-Window Transformers without PE Remain Turing Complete

  • 提出HIST模型,用窗口词频统计实现自回归生成
  • 证明无位置编码的滑动窗口变压器可模拟图灵机
  • 适合研究模型表达能力与序列建模本质的研究者

位置编码(PE)常被视为Transformer处理有序序列的必要条件:无PE时,其上下文对后续词的映射呈现置换不变性。这一直是以往通用性结果的基础,依赖位置信息证明带思维链的Transformer可执行任意计算,即具备图灵完备性。本文聚焦长文本推理场景,其中生成通过固定大小的滑动上下文窗口进行。我们发现,窗口机制本身(轻微地)打破了置换对称性。为精确刻画此额外表达能力,引入抽象自回归模型HIST:每次更新仅依赖常数大小内部状态和当前窗口内的词频直方图。我们证明该模型图灵完备,因窗口演化能揭示刚离开窗口的词,足以模拟图灵完备的Post机器。随后构建一个无位置编码、基于常数大小词表的滑动窗口Transformer,并证明其可模拟HIST模型。结果表明:位置编码并非实现通用计算的必需品;窗口滑动本身已打破置换对称性并捕获足够位置信息。

原文摘要 · Abstract (English)

Positional encoding (PE) is widely viewed as necessary for transformers to process ordered sequences: without them, the next-token map appears permutation-invariant in its context tokens. This intuition underlies all prior universality results, which rely on positional information to prove that transformers with chain-of-thought can perform arbitrary computation, i.e., they are Turing complete. We revisit this belief in the regime most relevant to long-form reasoning, where generation proceeds through a finite sliding context window. Our opening perception is that the window mechanism itself (mildly) breaks the permutation symmetry. To distill and precisely capture the degree of this added expressiveness, we introduce an abstract autoregressive model, the HIST model, in which each update depends only on constant-size internal state and the token-count histogram within the current window. We prove that this HIST model is Turing complete by showing that the evolution of the window can reveal the token that has just left the window, which suffices to simulate Turing-complete Post machines. We then construct a sliding-window transformer over a constant-size token alphabet, without PE, and show that it can simulate the HIST model. Our result demonstrates that positional encodings are not indispensable for transformers to perform universal computation: The window sliding itself already breaks permutation symmetry and captures sufficient positional information.

Transformer图灵完备序列建模位置编码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。