解析位置编码对Transformer表达与泛化的影响，提出更优的正交函数编码方法。
Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalization
- 基于函数逼近理论分析不同位置编码的表达能力
- 正交变换编码在泛化与长序列外推上优于正弦编码
- 为自然语言与视觉任务中的编码设计提供理论依据
位置编码是Transformer处理序列数据的核心组件。本文构建理论框架，分析正弦、可学习、相对及基于偏置的方法（如ALiBi）对模型表达能力、泛化性能和长序列外推能力的影响。通过函数逼近分析表达能力，利用Rademacher复杂度建立泛化界，并提出基于小波与勒让德多项式的正交函数编码方法。实验在合成序列到序列任务中显示，正交变换编码在泛化与外推能力上优于传统正弦编码。该研究填补了Transformer理论空白，为自然语言处理、计算机视觉等领域的编码设计提供理论支持。
原文摘要 · Abstract (English)
Positional encodings are a core part of transformer-based models, enabling processing of sequential data without recurrence. This paper presents a theoretical framework to analyze how various positional encoding methods, including sinusoidal, learned, relative, and bias-based methods like Attention with Linear Biases (ALiBi), impact a transformer's expressiveness, generalization ability, and extrapolation to longer sequences. Expressiveness is defined via function approximation, generalization bounds are established using Rademacher complexity, and new encoding methods based on orthogonal functions, such as wavelets and Legendre polynomials, are proposed. The extrapolation capacity of existing and proposed encodings is analyzed, extending ALiBi's biasing approach to a unified theoretical context. Experimental evaluation on synthetic sequence-to-sequence tasks shows that orthogonal transform-based encodings outperform traditional sinusoidal encodings in generalization and extrapolation. This work addresses a critical gap in transformer theory, providing insights for design choices in natural language processing, computer vision, and other transformer applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。