用随机浮点位置编码提升Transformer对长文本的泛化能力
Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers
- 用连续随机值代替离散位置索引，避免长序列外推时的分布外问题
- 在未见长度上表现更优，零样本常识推理任务准确率显著提升
- 可无缝融入现有位置编码，如RoPE、ALiBi，适配性强
长度泛化指语言模型在训练时未见过的更长输入上保持性能的能力。本文提出一种简单而有效的位置编码策略——随机浮点采样（RFS），能良好泛化到预训练或微调阶段未见的长度。RFS不从预定义的离散位置集合中选取索引，而是使用随机采样的连续值，从而在训练中暴露模型于多样化的索引，避免在未见长度上出现分布外（OOD）问题。由于位置索引分配是主流位置编码中的基础步骤，RFS可轻松集成至绝对正弦编码、RoPE和ALiBi等方法中。实验表明，RFS在长度泛化任务及零样本常识推理基准上均取得更优性能。
原文摘要 · Abstract (English)
Length generalization is the ability of language models to maintain performance on inputs longer than those seen during pretraining. In this work, we introduce a simple yet powerful position encoding (PE) strategy, Random Float Sampling (RFS), that generalizes well to lengths unseen during pretraining or fine-tuning. In particular, instead of selecting position indices from a predefined discrete set, RFS uses randomly sampled continuous values, thereby avoiding out-of-distribution (OOD) issues on unseen lengths by exposing the model to diverse indices during training. Since assigning indices to tokens is a common and fundamental procedure in widely used PEs, the advantage of RFS can easily be incorporated into, for instance, the absolute sinusoidal encoding, RoPE, and ALiBi. Experiments corroborate its effectiveness by showing that RFS results in superior performance in length generalization tasks as well as zero-shot commonsense reasoning benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。