多层自回归Transformer无需显式位置编码也能区分序列顺序。
Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph
- 多层Transformer通过堆叠无序处理模块，可自发产生序列敏感行为。
- 单层模型必须依赖位置编码，而多层模型可自行学习顺序信息。
- 适合关注模型本质机制的开发者与研究者阅读。
自回归Transformer语言模型是否需要显式的位置编码（PEs）？答案是：只要层数超过一层，就不需要——它们能区分经过置换的词序，而无需显式位置编码。这是因为一系列（对排列不变的）集合处理器级联后，在自回归设定下可集体表现出序列敏感性。这一性质早在GPT-2同期的研究中已被发现，但未被广泛传播，导致近期出现重复发现。这可能与GPT-2/3发布后语言建模社区的迅速扩张有关，也可能因早期文献缺乏清晰解释所致。本文回顾这一被遗忘的原理及其起源，旨在重新确立其作为通用知识的地位。
原文摘要 · Abstract (English)
Do autoregressive Transformer language models require explicit positional encodings (PEs)? The answer is 'no' provided they have more than one layer -- they can distinguish sequences with permuted tokens without the need for explicit PEs. This follows from the fact that a cascade of (permutation invariant) set processors can collectively exhibit sequence-sensitive behavior in the autoregressive setting. This property has been known since early efforts (contemporary with GPT-2) adopting the Transformer for language modeling. However, this result does not appear to have been well disseminated, leading to recent rediscoveries. This may be partially due to a sudden growth of the language modeling community after the advent of GPT-2/3, but perhaps also due to the lack of a clear explanation in prior work, despite being commonly understood by practitioners in the past. Here we review the long-forgotten explanation why explicit PEs are nonessential for multi-layer autoregressive Transformers (in contrast, one-layer models require PEs to discern order information of their inputs), as well as the origin of this result, and hope to re-establish it as a common knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。