提出可外推的精确位置编码，让模型处理更长序列时更准确。
ExPe: Exact Positional Encodings for Generative Transformer Models with Extrapolating Capabilities
- 通过覆盖嵌入向量特定维度实现精确位置编码
- 在长于训练序列的文本上显著降低困惑度
- 适合需要处理超长序列的任务，如长文本生成
本文提出一种新型位置嵌入方法——精确位置编码（ExPE），可在绝对位置嵌入基础上实现对更长序列的外推能力。传统Transformer模型依赖绝对或相对位置编码，难以泛化到训练时未见的长序列。ExPE通过覆盖嵌入向量的特定维度，精确编码位置信息，在保持原嵌入完整性的同时提升模型对更长序列的泛化能力。在因果语言建模任务中，当测试序列长度超过训练序列时，ExPE相比旋转编码（rotary）和正弦编码（sinusoidal）显著降低困惑度。
原文摘要 · Abstract (English)
This paper introduces a novel approach to position embeddings in transformer models, named "Exact Positional Embeddings" (ExPE). An absolute positional embedding method that can extrapolate to sequences of lengths longer than the ones it was trained on. Traditional transformer models rely on absolute or relative position embeddings to incorporate positional information into token embeddings, which often struggle with extrapolation to sequences longer than those seen during training. Our proposed method utilizes a novel embedding strategy that encodes exact positional information by overriding specific dimensions of the embedding vectors, thereby enabling a more precise representation of token positions. The proposed approach not only maintains the integrity of the original embeddings but also enhances the model's ability to generalize to more extended sequences. In causal language modeling, our ExPE embeddings significantly reduce perplexity compared to rotary and sinusoidal embeddings, when tested on sequences longer than those used in training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。