通过搜索轨迹生成高混淆干扰项,用细粒度评分提升长文本推理能力
LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards

- 用知识图谱生成多跳问题,结合搜索轨迹构建高低混淆干扰文档
- 设计实体级评分机制,仅对正确答案进行过程监督,避免奖励作弊
- 在5个基准上验证,3种规模模型均显著优于基线,适合提升推理可信度
长文本推理仍是大语言模型的核心挑战,常因无法定位和整合关键信息而失效。现有基于可验证奖励的强化学习方法受限于低混淆干扰项和稀疏的结果导向奖励信号,难以指导中间推理步骤。为此,本文提出LongTraceRL。数据构建方面,通过知识图谱随机游走生成多跳问题,并利用搜索代理轨迹构建分层干扰项:代理读取但未引用的文档(高混淆)、出现在搜索结果中却从未打开的文档(低混淆),使训练上下文远比随机采样或单次搜索更难。奖励设计方面,提出一种评分器奖励,以每条推理链上的黄金实体作为细粒度、实体级别的过程监督。该奖励仅作用于最终答案正确的响应(正向仅策略),可区分正确回答间的推理质量,防止奖励劫持。在三个推理类LLM(4B–30B)和五个长上下文基准上的实验表明,LongTraceRL持续优于强基线,能有效促进全面且证据充分的推理。代码、数据集和模型已开源。
原文摘要 · Abstract (English)
Long-context reasoning remains a central challenge for large language models, which often fail to locate and integrate key information in extensive distracting content. Reinforcement learning with verifiable rewards (RLVR) has shown promise for this task, yet existing methods are limited by low-confusability distractors and sparse, outcome-only reward signals that cannot supervise intermediate reasoning steps. To address these issues, we introduce \textsc{LongTraceRL}. For data construction, we generate multi-hop questions via knowledge graph random walks and leverage search agent trajectories to build \emph{tiered distractors}: documents the agent read but did not cite (high confusability) and documents that appeared in search results but were never opened (low confusability), producing training contexts that are far more challenging than those built by random sampling or one-shot search. For reward design, we propose a \emph{rubric reward} that uses the gold entities along each reasoning chain as fine-grained, entity-level process supervision. This rubric reward is applied only to responses with correct final answers (positive-only strategy), distinguishing the reasoning quality among correct responses and preventing reward hacking. Experiments on three reasoning LLMs (4B--30B) across five long-context benchmarks demonstrate that \textsc{LongTraceRL} consistently outperforms strong baselines and encourages comprehensive, evidence-grounded reasoning. Codes, datasets and models are available at \href{https://github.com/THU-KEG/LongTraceRL}{https://github.com/THU-KEG/LongTraceRL}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。