提出统一框架与自适应位置编码,实现长文本无限外推。
Towards Infinite Length Extrapolation: A Unified Approach
- 将位置编码重构为乘性变换与加性偏置的分解,统一现有方法
- 在32,000词长故事上保持稳定性能,突破传统上下文限制
- 适合需要处理超长序列的模型研究者与工业应用开发者
大语言模型虽已革新自然语言处理,但其序列处理能力受限于训练时的上下文窗口大小。现有长度外推方法常伴随性能下降或计算效率问题。本文提出统一框架,将位置编码视为注意力分数的乘性变换与加性偏置之分解,不仅涵盖相对位置嵌入和注意力偏置调控等主流方法,还揭示其在远距离依赖建模上的固有缺陷。基于此,提出自适应位置编码(APE),融合自适应频率调制与包含线性、对数及平方根项的精细衰减偏置。理论分析表明,在无界序列下仍可保证Softmax归一化合理,维持长程相关性、熵有界性和梯度位置敏感性。实验在TinyStories数据集及新构建的最长达32,000词的合成数据集《Long Tiny Stories》上验证了有效性。相关代码、数据集与模型权重见https://anonymous.4open.science/r/Check-2DAD/
原文摘要 · Abstract (English)
Large language models (LLMs) have revolutionized natural language processing, but their ability to process long sequences is fundamentally limited by the context window size during training. Existing length extrapolation methods often suffer from performance degradation or computational inefficiencies. We thereby use a unified framework that reinterprets positional encoding methods as a decomposition of the attention score into a multiplicative transformation and an additive bias. This perspective not only subsumes popular approaches such as relative position embeddings and attention-bias moderated approaches but also exposes their inherent limitations in handling long-range dependencies. To address these shortcomings, motivated by our framework, we introduce Adaptive Positional Encoding (APE), which leverages adaptive frequency modulation and an intricately designed decay bias that incorporates linear, logarithmic, and square-root terms. Our theoretical analysis establishes conditions for infinite-context extrapolation, ensuring that the softmax normalization remains well-defined over unbounded sequences while preserving long-distance correlations, entropy boundedness and gradient positional sensitivity. We substantiate our claims with an experimental case study on TinyStories dataset as well as a new synthetic dataset, \emph{Long Tiny Stories} featuring stories up to 32,000 words. Relevant code, dataset and model weights are available at https://anonymous.4open.science/r/Check-2DAD/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。