arXiv:2507.19786cs.CL2025-07AAAI被引 2

无需人工或模型干预,任意拼接指令生成长文本上下文。

Flora: Effortless Context Construction to Arbitrary Length and Scale

  • 用分类指令拼接生成长上下文,无需人工或模型参与。
  • 在Llama3-8B-Instruct和QwQ-32B上显著提升长上下文表现。
  • 保持短上下文性能几乎不变,适合长文本任务研究者。

长上下文处理对大语言模型(LLM)构成挑战,源于长文本稀缺、计算开销高及短上下文能力严重退化。现有方法需依赖人类或大模型构建长上下文,成本高且长度与多样性受限。同时,当前长上下文模型的短上下文性能下降明显。本文提出无须人工或大模型参与的长上下文构建策略Flora。Flora通过类别化短指令任意拼接,并以元指令引导模型生成响应,实现任意长度与规模的丰富多样长上下文生成,仅轻微影响短上下文性能。在Llama3-8B-Instruct和QwQ-32B上的实验表明,增强后的模型在三个长上下文基准测试中表现优异,同时保持强短上下文能力。数据构建代码已开源。

原文摘要 · Abstract (English)

Effectively handling long contexts is challenging for Large Language Models (LLMs) due to the rarity of long texts, high computational demands, and substantial forgetting of short-context abilities. Recent approaches have attempted to construct long contexts for instruction tuning, but these methods often require LLMs or human interventions, which are both costly and limited in length and diversity. Also, the drop in short-context performances of present long-context LLMs remains significant. In this paper, we introduce Flora, an effortless (human/LLM-free) long-context construction strategy. Flora can markedly enhance the long-context performance of LLMs by arbitrarily assembling short instructions based on categories and instructing LLMs to generate responses based on long-context meta-instructions. This enables Flora to produce contexts of arbitrary length and scale with rich diversity, while only slightly compromising short-context performance. Experiments on Llama3-8B-Instruct and QwQ-32B show that LLMs enhanced by Flora excel in three long-context benchmarks while maintaining strong performances in short-context tasks. Our data-construction code is available at \href{https://github.com/txchen-USTC/Flora}{https://github.com/txchen-USTC/Flora}.

长上下文指令拼接零样本构造模型增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。