无需位置编码，因果Transformer通过嵌入相似性自动捕捉位置信息
Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
- 利用相邻嵌入更相似的特性，隐式建模位置关系
- 无论随机初始化或训练后，该模式在多数超参下均成立
- 适用于希望简化结构的NLP任务开发者
使用因果注意力机制的Transformer可在不使用位置编码的情况下解决需要位置信息的任务。本文提出并检验了一种新假设：位置信息可通过嵌入之间的相似性隐式存储。我们观察到，在具有因果注意力且无位置编码的Transformer中，邻近词元的嵌入比远距离词元更相似，这使得模型可能重构词元位置。该现象在训练后及随机初始化的模型中，于常见超参数范围内均出现。
原文摘要 · Abstract (English)
Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。