用自生成方法批量制造高质量长文本指令数据，解决训练难题。
LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
- 让对齐的LLM读文档后自动生成相关问题，形成问答对。
- 在三个评测集上表现领先，且短文本任务也不逊色。
- 无需人工标注，适合需要大规模长文本训练的研究者。
高质量的长上下文指令数据对于对齐长上下文大语言模型至关重要。尽管Qwen和Llama等模型已公开，但其长上下文指令数据仍属私有。人工标注成本高且困难，模板化合成方法则限制了规模、多样性和质量。我们提出LongMagpie，一种自合成框架，可自动生成大规模长上下文指令数据。核心思路是：当对齐的长上下文LLM被给予一段文档及特殊标记符后，会自回归生成与上下文相关的查询。通过收集这些文档-查询对及其模型回答，LongMagpie实现了无须人工参与的高质量指令生成。在HELMET、RULER和Longbench v2上的实验表明，LongMagpie在长上下文任务中达到领先性能，同时在短上下文任务上也保持竞争力，证明其是开放、多样且可扩展的长上下文指令数据合成的有效方法。
原文摘要 · Abstract (English)
High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。