通过随机位置编码提升模型对超长文本的推理能力
Randomized YaRN Improves Length Generalization for Long-Context Reasoning
- 在短文本训练中引入大范围随机位置编码，模拟长序列分布
- 在16K至128K长度上均显著提升推理表现，远超标准微调
- 适合需要处理超长上下文的任务，如复杂问答与多轮指代
大型语言模型通常在短序列上预训练，再通过额外训练扩展到更长序列。然而，这些模型在进一步推广到极长序列时仍存在困难。我们提出随机化YaRN（Randomized YaRN），一种结合基于YaRN的位置外推、随机位置编码和长度课程的学习方法。在短上下文数据训练期间，将来自更大位置范围的YaRN位置编码随机分配给令牌，使模型即使在短输入中也接触到分布外的位置表示。我们在三个挑战性的长上下文推理基准测试中评估了该方法：BABILong、多轮指代消解（MRCR）和LongBench v2。当在短上下文数据上训练时，随机化YaRN在16K至128K长度上的推理性能持续优于标准微调，尤其在远超出分布的长度上提升最显著。结果表明，逐步暴露模型于分布外的位置分布是一种实现可泛化长上下文推理的有效策略。
原文摘要 · Abstract (English)
Large language models (LLMs) are typically pretrained on short sequences and then extended to work on longer sequences with additional training. However, such LLMs still struggle to further generalize to very long sequences. We propose Randomized YaRN, a training method that improves length generalization by combining YaRN-based positional extrapolation with randomized positional encoding and a length curriculum. During training on short context data, tokens are assigned YaRN positional encodings sampled from a larger position range, exposing the model to out-of-distribution positional representations even on short-context inputs. We evaluate Randomized YaRN on three challenging long-context reasoning benchmarks, BABILong, Multi-Round Coreference Resolution (MRCR), and LongBench v2. When training on data with short context, Randomized YaRN consistently improves reasoning performance on context lengths from 16K to 128K and outperforms standard fine-tuning, with the largest gains appearing at far out-of-distribution lengths. Our results suggest that progressively exposing models to OOD positional distributions provides an effective recipe for generalizable long-context reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。