用高效训练法把大模型上下文从8k扩展到128k，仅需单卡一天。
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models
- 通过关键标记分析与位置编码转换，模拟长序列输入
- 仅用30%目标窗口大小，降低85%以上训练开销
- 开源模型可达接近GPT-4的长文本理解能力
大型语言模型（LLMs）在处理长上下文任务时面临挑战，因其预训练阶段的有效上下文窗口有限，难以泛化于长序列。而通过后训练扩展上下文窗口成本高昂。为此，本文提出LongRecipe，一种高效的上下文窗口扩展训练策略，包含关键标记分析、位置索引变换和训练优化方法。该方法在保持训练效率的同时显著提升模型对长程依赖的理解能力。在三类LLM上的实验表明，LongRecipe可在仅需目标窗口30%长度的情况下利用长序列，并使计算资源消耗减少超过85%。同时，原模型在通用任务中的能力得以保留。最终，可将开源LLM的有效上下文窗口从8k扩展至128k，仅用一天专用训练即达成接近GPT-4的性能，使用单张80GB显存的GPU即可完成。
原文摘要 · Abstract (English)
Large language models (LLMs) face significant challenges in handling long-context tasks because of their limited effective context window size during pretraining, which restricts their ability to generalize over extended sequences. Meanwhile, extending the context window in LLMs through post-pretraining is highly resource-intensive. To address this, we introduce LongRecipe, an efficient training strategy for extending the context window of LLMs, including impactful token analysis, position index transformation, and training optimization strategies. It simulates long-sequence inputs while maintaining training efficiency and significantly improves the model's understanding of long-range dependencies. Experiments on three types of LLMs show that LongRecipe can utilize long sequences while requiring only 30% of the target context window size, and reduces computational training resource over 85% compared to full sequence training. Furthermore, LongRecipe also preserves the original LLM's capabilities in general tasks. Ultimately, we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with 80G memory. Our code is released at https://github.com/zhiyuanhubj/LongRecipe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。