用文档重建训练大模型长文本理解能力,无需人工标注。
Document Reconstruction Unlocks Scalable Long-Context RLVR
- 通过替换段落为占位符,让模型学习重建完整文档。
- 在RULER上显著提升性能,在LongBench v2上也获合理改进。
- 适合想低成本提升长文本推理能力的研究者使用。
强化学习结合可验证奖励(RLVR)已成为提升大语言模型(LLMs)长文本能力的主流方法。然而,该方法通常依赖于强大教师模型或人类专家提供的标准答案或评估标准,成本高且耗时。本文探索无监督方法,以消除对人工标注或教师模型监督的需求。具体而言,我们在长文档中用特殊占位符替换部分段落,通过强化学习训练模型从候选选项中正确识别并排序缺失段落以重建原文。这一训练范式使模型能够捕捉全局叙事连贯性,显著提升长文本处理能力。我们在RULER和LongBench v2两个常用基准上验证了该方法的有效性。尽管在RULER上取得明显增益,且在无需人工长文本问答数据的情况下,也能在LongBench v2上实现合理提升。此外,我们进行了广泛的消融实验,分析奖励设计、数据构建策略、训练方案及数据规模对模型表现的影响。代码、数据与模型均已公开。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards~(RLVR) has become a prominent paradigm to enhance the capabilities (i.e.\ long-context) of Large Language Models~(LLMs). However, it often relies on gold-standard answers or explicit evaluation rubrics provided by powerful teacher models or human experts, which are costly and time-consuming. In this work, we investigate unsupervised approaches to enhance the long-context capabilities of LLMs, eliminating the need for heavy human annotations or teacher models' supervision. Specifically, we first replace a few paragraphs with special placeholders in a long document. LLMs are trained through reinforcement learning to reconstruct the document by correctly identifying and sequencing missing paragraphs from a set of candidate options. This training paradigm enables the model to capture global narrative coherence, significantly boosting long-context performance. We validate the effectiveness of our method on two widely used benchmarks, RULER and LongBench~v2. While acquiring noticeable gains on RULER, it can also achieve a reasonable improvement on LongBench~v2 without any manually curated long-context QA data. Furthermore, we conduct extensive ablation studies to analyze the impact of reward design, data curation strategies, training schemes, and data scaling effects on model performance. We publicly release our code, data, and models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。