通过精细奖励设计,让大模型在长文本推理中更深入思考,提升准确率。
Incentivizing In-depth Reasoning over Long Contexts with Process Advantage Shaping
- 构建高难度多跳问答对,引导模型深度推理
- 识别并保留接近正确的推理路径中的学习信号
- 小模型也能达到顶尖大模型的长文本推理水平
强化学习结合可验证奖励(RLVR)在短文本推理中表现优异,但在需要精确定位和长程推理的长文本场景中性能下降。我们发现长文本推理存在‘差之毫厘’现象:推理路径基本正确,但最终答案错误。原因有二:一是长文本问答数据中缺乏高推理密度,难以推动模型进行复杂多跳推理;二是长文本强化学习训练中,对部分正确但结果错误的轨迹进行了无差别惩罚,导致重要学习信号丢失。为此,我们提出 DeepReasonQA,一个基于知识图谱的合成框架,可控生成具有内在推理链的高难度多跳长文本问答对。在此基础上,提出长文本过程优势塑造(LongPAS),通过有效性与相关性双维度评估推理步骤,实现细粒度信用分配,捕捉‘差之毫厘’轨迹中的关键学习信号。在三个长文本推理基准上的实验表明,该方法显著优于RLVR基线,且仅用极少参数即达到前沿大模型水平。进一步分析证实其有效增强长文本推理能力,并保持强化学习训练的稳定性。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing LLMs short-context reasoning, but its performance degrades in long-context scenarios that require both precise grounding and robust long-range reasoning. We identify the "almost-there" phenomenon in long-context reasoning, where trajectories are largely correct but fail at the final step, and attribute this failure to two factors: (1) the lack of high reasoning density in long-context QA data that push LLMs beyond mere grounding toward sophisticated multi-hop reasoning; and (2) the loss of valuable learning signals during long-context RL training due to the indiscriminate penalization of partially correct trajectories with incorrect outcomes. To overcome this bottleneck, we propose DeepReasonQA, a KG-driven synthesis framework that controllably constructs high-difficulty, multi-hop long-context QA pairs with inherent reasoning chains. Building on this, we introduce Long-context Process Advantage Shaping (LongPAS), a simple yet effective method that performs fine-grained credit assignment by evaluating reasoning steps along Validity and Relevance dimensions, which captures critical learning signals from "almost-there" trajectories. Experiments on three long-context reasoning benchmarks show that our approach substantially outperforms RLVR baselines and matches frontier LLMs while using far fewer parameters. Further analysis confirms the effectiveness of our methods in strengthening long-context reasoning while maintaining stable RL training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。