位置编码让Transformer在词表上下文学习中具备函数逼近能力
Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding
- 单层Transformer无位置编码时无法实现通用逼近
- 加入位置编码后，模型可逼近任意函数，满足通用逼近性质
- 为位置编码在上下文学习中的理论作用提供新解释
大量研究已证明Transformer具备上下文学习（ICL）能力。在函数逼近场景中，上下文可作为控制参数，使模型具备通用逼近性质（UAP）。实际中，上下文由有限词汇表中的标记表示，即本文所关注的词表上下文学习（VICL）。我们证明：单层Transformer在无位置编码时不具备UAP；但引入位置编码后，可实现UAP。论文进一步给出了若干位置编码的充分条件。研究揭示了位置编码在上下文学习中从逼近论角度带来的优势。
原文摘要 · Abstract (English)
Numerous studies have demonstrated that the Transformer architecture possesses the capability for in-context learning (ICL). In scenarios involving function approximation, context can serve as a control parameter for the model, endowing it with the universal approximation property (UAP). In practice, context is represented by tokens from a finite set, referred to as a vocabulary, which is the case considered in this paper, \emph{i.e.}, vocabulary in-context learning (VICL). We demonstrate that VICL in single-layer Transformers, without positional encoding, does not possess the UAP; however, it is possible to achieve the UAP when positional encoding is included. Several sufficient conditions for the positional encoding are provided. Our findings reveal the benefits of positional encoding from an approximation theory perspective in the context of ICL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。