用分层合成数据让大模型支持百万级上下文，突破长文本处理瓶颈
Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
- 通过分层合成数据生成扩展上下文长度，无需依赖真实长文本标注
- 模型支持100万词符上下文，在RULER和InfiniteBench上表现优异
- 适合需要处理超长文档的场景，如法律、科研文献分析
大语言模型在长上下文推理方面面临挑战，不仅因序列长度增加导致计算复杂度呈二次增长，还受限于长文本数据稀缺且标注成本高。目前几乎无开源工作系统性分析长上下文数据，也缺乏超过10万词符的公开指令微调数据集。为此，我们提出一种新型后训练合成数据生成策略，可高效扩展大模型的上下文窗口，同时保持其通用任务性能。该方法可无限扩展至任意长上下文，不受真实数据长度限制，有效缓解原始长文本数据不足问题。通过逐步旋转位置编码（RoPE）缩放训练策略，我们证明模型在长达100万词符的上下文下，仍能在RULER基准和InfiniteBench上表现良好，并维持对通用语言任务的稳健性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) struggle with long-context reasoning, not only due to the quadratic scaling of computational complexity with sequence length but also because of the scarcity and expense of annotating long-context data. There has been barely any open-source work that systematically ablates long-context data, nor is there any openly available instruction tuning dataset with contexts surpassing 100K tokens. To bridge this gap, we introduce a novel post-training synthetic data generation strategy designed to efficiently extend the context window of LLMs while preserving their general task performance. Our approach scalably extends to arbitrarily long context lengths, unconstrained by the length of available real-world data, which effectively addresses the scarcity of raw long-context data. Through a step-by-step rotary position embedding (RoPE) scaling training strategy, we demonstrate that our model, with a context length of up to 1M tokens, performs well on the RULER benchmark and InfiniteBench and maintains robust performance on general language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。