arXiv:2607.06160cs.CLcs.AI2026-07

用证据图引导指令生成,提升大模型长文本理解能力。

LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis

论文配图:LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis
图 1 · 摘自论文原文
  • 构建分层任务体系与证据图,实现可控难度的长文本指令合成。
  • 在LongBench等数据集上超越所有基线模型,高难度任务提升显著。
  • 适合需要精准长文推理、避免中间信息丢失的研究者使用。

合成长上下文监督微调(SFT)数据是提升大语言模型长文本理解能力的可扩展方法,但现有方法存在任务覆盖窄、指令难度不足和缺乏忠实度监督三大局限。本文提出LongCrafter,一种结构化合成框架,结合分层任务分类体系与证据基础生成流程。该分类体系将长文本理解划分为局部/浅层与全局/深层两个层次,生成32种细粒度任务类型作为全局生成先验。基于此,LongCrafter构建对齐任务的长上下文,将其分解为显式证据图以建模段落间依赖关系,并生成严格基于定位证据片段的指令-响应对,确保难度可控且推理可追溯。在Qwen2.5-7B与LLaMA-3.1-8B模型上微调后,其性能超越所有SFT基线,甚至优于官方后续训练模型,在LongBench、LongBench~v2和LooGLE上表现优异,尤其在高难度任务中提升最明显。进一步分析表明,LongCrafter数据更具多样性,难度分布更均衡,且训练模型能鲁棒地定位证据,无论位置如何,有效缓解了‘中间丢失’问题。

原文摘要 · Abstract (English)

Synthesizing long-context supervised fine-tuning (SFT) data is a scalable way to enhance the long-context understanding of large language models (LLMs), yet existing approaches share three limitations: narrow task coverage, insufficient instruction difficulty, and a lack of faithfulness supervision. We propose \textbf{LongCrafter}, a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline. The taxonomy organizes long-context understanding into local/shallow and global/deep levels and yields 32 fine-grained task types that serve as a global generative prior. Guided by this taxonomy, LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans, ensuring both controllable difficulty and faithful, traceable reasoning. Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models on LongBench, LongBench~v2, and LooGLE across both Qwen2.5-7B and LLaMA-3.1-8B, with the largest gains on high-difficulty tasks. Further analysis shows that LongCrafter data is more diverse and better spread across difficulty levels, and that the trained models locate evidence robustly regardless of position, effectively mitigating the ``lost in the middle'' problem.

长文本理解指令合成证据图大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。