arXiv:2606.17029cs.CL2026-06被引 2

用证据树生成更精准的评分标准,让研究型AI训练更快更准。

DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents

论文配图:DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents
图 1 · 摘自论文原文
  • 反向构建:先定评估目标,再生成匹配的问答对。
  • 9000组数据训练,节省13倍强化学习算力。
  • 适合想高效训练长文生成AI的研究者。

深度研究代理通过检索和推理证据来生成长篇报告。基于评分标准的强化学习能通过可验证的标准将报告质量转化为奖励信号,从而优化模型,但其效率取决于这些标准是否准确覆盖任务范围与证据需求。现有方法多由大模型为给定查询生成评分标准,当模型无法推断底层信息需求时,生成的评分标准可能不完整,降低强化学习效率。为此,我们提出DeepRubric,一种逆向数据构建框架:不从查询推导评价标准,而是先确定有证据支持的报告应评估哪些方面,再基于这些评估目标合成对齐的查询-评分对。从采样的种子主题出发,DeepRubric通过递归扩展有证据支持的子问题构建证据树,其叶节点作为原子且可验证的评估目标。随后利用该证据树合成训练用查询与评分标准,确保奖励信号精确对应查询所要求的信息。使用DeepRubric构建了9000组查询-评分监督样本,并基于评分标准采用GRPO训练DeepRubric-8B,在三个基准上达到与先前开源最先进模型相当的性能,但仅需约1/13的强化学习GPU小时。

原文摘要 · Abstract (English)

Deep research agents synthesize long-form reports by searching and reasoning over retrieved evidence. Reinforcement learning with rubric-based rewards improves these agents by optimizing them against checkable criteria that translate report quality into reward signals, but its efficiency depends on whether those criteria reliably capture the task scope and evidence needs. Most existing studies ask an LLM to generate rubrics for a given query, but when the model fails to infer the underlying information needs, the generated rubrics may be incomplete and reduce RL efficiency. To obtain more reliable query--rubric supervision, we introduce DeepRubric, a data construction framework that reverses this process: instead of inferring evaluation criteria for a given query, it first determines what an evidence-backed report should be evaluated on and then synthesizes aligned query--rubric pairs from those evaluation targets. Starting from a sampled seed topic, DeepRubric builds an evidence tree by recursively expanding evidence-backed sub-questions, whose leaves serve as atomic and verifiable evaluation targets. It then uses the evidence tree to synthesize the training query and rubrics, ensuring that the reward evaluates exactly the information requested by the query. Using DeepRubric, we construct 9K query--rubric supervision examples and train DeepRubric-8B with rubric-based GRPO, achieving comparable performance to prior open state-of-the-art deep research models across three benchmarks with roughly 13x fewer RL GPU-hours.

强化学习研究代理评分标准高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。