无需位置编码，Transformer仍可高效识别生成层级语言。
Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding
- 用起始符和因果掩码隐式获取位置与深度信息。
- 模型在不依赖显式位置编码下成功生成层级语言。
- 位置编码可能损害模型对序列长度的泛化能力。
本研究通过构造性证明表明，即使不使用特定的位置编码，Transformer也能以与模型规模成比例的效率识别和生成层级语言。具体而言，我们证明了因果掩码与起始标记能够使Transformer自主计算出层级结构中的位置信息与深度。实验表明，无需显式位置编码的Transformer仍可生成层级语言。此外，研究提出显式位置编码可能对模型在序列长度上的泛化性能产生负面影响。
原文摘要 · Abstract (English)
In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding. Specifically, we show that causal masking and a starting token enable Transformers to compute positional information and depth within hierarchical structures. We demonstrate that Transformers without positional encoding can generate hierarchical languages. Furthermore, we suggest that explicit positional encoding might have a detrimental effect on generalization with respect to sequence length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。