arXiv:2603.02146cs.CL2026-03被引 16

让大模型在长文本中精准找证据,提升推理能力。

LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards

  • 用可验证的上下文奖励替代仅看答案的奖励,引导模型找对信息。
  • 在RULER-QA和LongBench v2上,14B模型准确率提升至88.90和46.5。
  • 适合需要长期依赖、精准引用的复杂推理任务使用。

基于可验证奖励的强化学习(RLVR)通过优化大语言模型(LLM)以符合事实结果,显著提升了其推理能力。然而,在长上下文场景下,该范式因依赖内部参数化知识而失效,无法有效处理需外部信息支撑的任务——即上下文定位能力。我们发现根本原因在于:仅以最终答案为奖励信号过于稀疏,难以指导模型识别相关证据。我们正式证明,仅基于结果的奖励会导致上下文定位过程梯度消失,使学习不可行。为此,我们提出LongRLVR,引入密集且可验证的上下文奖励作为辅助信号,直接激励模型选择正确依据,提供稳定的学习梯度,解决核心优化难题。我们在Qwen与LLaMA模型上验证方法,覆盖多个长上下文基准。LongRLVR在所有模型和任务中均显著优于标准RLVR,例如将14B模型在RULER-QA上的得分从73.17提升至88.90,在LongBench v2上从39.8提升至46.5。结果表明,显式奖励上下文定位是释放大模型长文本推理潜力的关键策略。代码已开源。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) by optimizing them against factual outcomes. However, this paradigm falters in long-context scenarios, as its reliance on internal parametric knowledge is ill-suited for tasks requiring contextual grounding--the ability to find and reason over externally provided information. We identify a key reason for this failure: a reward based solely on the final answer is too sparse to effectively guide the model for identifying relevant evidence. We formally prove that the outcome-only reward leads to significant vanishing gradients for the context grounding process, rendering learning intractable. To overcome this bottleneck, we introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward. This auxiliary signal directly incentivizes the model for selecting the correct grounding information, providing a robust learning gradient that solves the underlying optimization challenge. We validate our method on challenging long-context benchmarks using Qwen and LLaMA models. LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks, e.g., boosting a 14B model's scores on RULER-QA from 73.17 to 88.90 and on LongBench v2 from 39.8 to 46.5. Our work demonstrates that explicitly rewarding the grounding process is a critical and effective strategy for unlocking the full reasoning potential of LLMs in long-context applications. Our code is available at https://github.com/real-absolute-AI/LongRLVR.

强化学习长文本推理可验证奖励大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。