将大模型上下文长度从128K扩展至400万tokens，保持性能平衡
From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
- 通过高效微调策略扩展上下文窗口，保留指令遵循能力
- 模型在长文本任务上达顶尖表现，短文本任务性能不降
- 适合需要超长上下文理解的文档、视频分析场景
长上下文能力对文档理解、视频分析、上下文学习和推理时扩展等应用至关重要，这些任务都需要模型处理和推理长序列文本与多模态数据。本文提出一种高效的训练方法，基于对齐的指令模型构建超长上下文大模型，将上下文长度从128K拓展至100万、200万和400万令牌。该方法利用高效的持续预训练策略扩展上下文窗口，并采用有效的指令微调保持模型的指令遵循与推理能力。基于Llama3.1-Instruct构建的UltraLong-8B模型在多种长上下文基准测试中表现领先。重要的是，该方法训练的模型在标准基准上仍保持竞争力，实现长/短上下文任务的均衡提升。我们还深入分析了关键设计选择，揭示了缩放策略与数据构成的影响。研究建立了高效扩展上下文长度的同时保持模型通用能力的可靠框架。所有模型权重已公开：https://ultralong.github.io/
原文摘要 · Abstract (English)
Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling, all of which require models to process and reason over long sequences of text and multimodal data. In this work, we introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach leverages efficient continued pretraining strategies to extend the context window and employs effective instruction tuning to maintain the instruction-following and reasoning abilities. Our UltraLong-8B, built on Llama3.1-Instruct with our recipe, achieves state-of-the-art performance across a diverse set of long-context benchmarks. Importantly, models trained with our approach maintain competitive performance on standard benchmarks, demonstrating balanced improvements for both long and short context tasks. We further provide an in-depth analysis of key design choices, highlighting the impacts of scaling strategies and data composition. Our findings establish a robust framework for efficiently scaling context lengths while preserving general model capabilities. We release all model weights at: https://ultralong.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。