提出新型位置编码函数，显著优于传统正弦编码。
Alternative positional encoding functions for neural transformers
- 设计了一组保留正弦特性但结构不同的周期函数用于位置编码
- 初步实验显示新方法在性能上大幅超越原始正弦编码
- 适合希望提升Transformer模型性能的研究者使用
神经Transformer深度架构中的关键模块是位置编码，它将位置信息以合适方式编码为Transformer神经层的输入。这一成功源于使用多种频率的正弦函数，以捕捉不同典型周期的重复模式。本文提出一组替代的周期函数用于位置编码，这些函数保留了正弦函数的一些关键特性，但在根本上与之不同。报告了一些初步实验结果，表明原始正弦版本被显著超越，强烈暗示这些替代函数可能在其他Transformer架构中具有更广泛的应用前景。
原文摘要 · Abstract (English)
A key module in neural transformer-based deep architectures is positional encoding. This module enables a suitable way to encode positional information as input for transformer neural layers. This success has been rooted in the use of sinusoidal functions of various frequencies, in order to capture recurrent patterns of differing typical periods. In this work, an alternative set of periodic functions is proposed for positional encoding. These functions preserve some key properties of sinusoidal ones, while they depart from them in fundamental ways. Some tentative experiments are reported, where the original sinusoidal version is substantially outperformed. This strongly suggests that the alternative functions may have a wider use in other transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。