让Transformer模型能准确处理比训练时长10倍的序列
Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
- 用概率化相对位置编码替代传统固定编码
- 在算术和组合任务上实现10倍长度外推
- 适合需要长序列推理的算法任务研究者
深度序列模型在测试序列远超训练长度时性能通常下降，而算法推理、多步计算和组合泛化等关键任务亟需强健的长度外推能力。我们提出PRISM，一种基于概率相对位置隐式叠加的新型位置编码机制，使Transformer模型可准确外推至训练长度的10倍。PRISM通过可微分直方图滤波更新学习连续相对位置，以概率叠加方式保留位置不确定性，而非传统确定性嵌入。实验表明，PRISM在算法基准测试中达到当前最优的长度外推性能，成功泛化到此前难以处理的序列长度，涵盖加法、乘法运算，SCAN组合性任务，以及来自DeepMind新数据集的复杂复制变体。分析显示，PRISM的概率位置编码能维持清晰可解释的内部状态，为可靠长度泛化提供理论支撑。这些成果推动了神经序列模型在远超训练范围下的算法鲁棒性目标。
原文摘要 · Abstract (English)
Deep sequence models typically degrade in accuracy when test sequences significantly exceed their training lengths, yet many critical tasks--such as algorithmic reasoning, multi-step arithmetic, and compositional generalization--require robust length extrapolation. We introduce PRISM, a Probabilistic Relative-position Implicit Superposition Model, a novel positional encoding mechanism that enables Transformers to extrapolate accurately up to 10x beyond their training length. PRISM learns continuous relative positions through a differentiable histogram-filter update, preserving position uncertainty via a probabilistic superposition rather than conventional deterministic embeddings. Empirically, PRISM achieves state-of-the-art length extrapolation, successfully generalizing to previously intractable sequence lengths across algorithmic benchmarks--including arithmetic (addition, multiplication), SCAN compositionality tasks, and complex copy variants derived from DeepMind's recent datasets. Our analysis demonstrates that PRISM's stochastic positional encoding maintains sharp and interpretable internal states, providing a theoretical basis for reliable length generalization. These results advance the goal of neural sequence models that remain algorithmically robust at lengths far exceeding their training horizon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。